. Hello everyone. In this video, we'll be building a multi-modal AI agent with RAC capabilities, leveraging NVIDIA's GPU accelerated models and open-source technologies. Let's start with a quick demo of our chatbot in action. As you can see, our chatbot can quickly process and respond to the question about the data provided. So, in this tutorial, we'll cover integrating a pipeline that accepts multimodal kind of data that includes text and vision. As part of the process, we'll cover how to set up vector database, generate embeddings, and how to use GPU-accelerated models. Now let's break down how we built this. Before we dive into the specifics of our chatbot, it's important to understand the ecosystem we are working with. NVIDIA provides a full- full stack ecosystem for large language models that integrate seamlessly with popular open source tools. This ecosystem offers flexibility and performance allowing developers to switch between different components as needed. NVIDIA not only provides proprietary tools but also contributes significantly to the open source community. Tools like Nemo for building and training large language models, Triton for inference serving, are prime examples of NVIDIA's contribution to open source. These tools integrate well with the broader AI ecosystem, providing developers with powerful GPU accelerated options for building LLM applications. NVIDIA Nemo goes beyond just a training framework. It includes open source components that enhance the capabilities for developers. Two notable open source components are Nemo Data Curator that simplifies the complex process of data curation. It helps extract, deduplicate, and filter information from large amounts of unstructured data, ensuring high quality, relevant data sets for training. The second component is Nemo Guard Rails. This component implements safety measures and controls for model output. It allows for developers to add guardrails that prevents inappropriate responses, enhancing the reliability and safety of AI application. developers to add guardrails that prevents inappropriate responses, enhancing the reliability and safety of AI application. You can learn more about this at github.com slash nvidia slash nemo. Let's quickly understand what RAG or retrieval augmented generation is. So RAG announces large language models by giving them access to external up-to-date knowledge. This process allows our chatbot to provide responses that are both general and specific to our patent data combining retrieval and generation for more accurate and contextual answers. Let's explore how we are extending RAG to handle various types of data or what we call a multimodal RAG pipeline. Now there are various ways to handle multimodal data, but in this case, we are grounding all the modality into a single one, which is text for this specific application. So we have introduced a specialized vision language model to process visual data. So in this case, one VLM is handling general images while the other vision language model is specifically tuned for charts and plots. Once the conversion of information is done, rest of the pipeline remains the same as a traditional rack pipeline. For our implementation of this application, we are using cutting edge tools from NVIDIA and the open source community. So let's break down the components of this specific pipeline. So to build our knowledge base, we first have to convert all the documents into text form. And to do that, we'll use vision language models as shared before. So for general image understanding, we'll be using Niva 22B model, which is NVIDIA's fine-tuned variant of Lava. variant of Lava. For charts and plots, we are employing dplot by Google, which is specialized in understanding graphical data, including charts and plots. For this specific example, we'll have some mix of documents, which is PDF and images that contain some of the blogs and slides from NVIDIA. So this data fuels our chatbot's knowledge base. For vector storage and similarity search, we'll use Milvus, a GPU-accelerated vector database that can handle large-scale similarity searches with impressive speed. So using GPU indexing and querying requires numerous query vectors or exceptionally high request pressure for maximum throughput and reduced latency compared to CPU indexing. For embedding, we'll use NV embed, which leverages NVIDIA's GPU acceleration to quickly transform text into high dimensional vectors. The GPU acceleration in both NV embed and Milvus significantly enhances the performance of our chatbot, allowing for faster indexing and querying of our knowledge base. For the actual question answering, we'll use a large language model, specifically LAMA3 from Meta, the 7TB instruct variant, which will be accessed through NVIDIA's NimAPI. So this gives us the power of state-of-the-art language models, which is GP optimized access through an API. Finally, we'll use Lama Index to orchestrate the entire process from query processing to retrieving relevant information and generating responses. We will also use Streamlit to create user-friendly interface for our chatbot where the user can directly upload or just provide the directory path of their documents and then simply query or ask question based on those documents. Now we have learned what the pipeline is about, now let's dive into the code. Before we dive into the individual script, let's take a quick look at our project structure. Our project consists of three main Python scripts. First one is app.py which is the main application file that sets up the streamlit interface and manages the overall flow of our multimodal rack system. streamlit interface and manages the overall flow of our multimodal rack system. Second is documentprocessors.py which is responsible for processing various documents types like PDFs, PowerPoint presentation, and images. Then comes the utils.py which contains utility functions for image processing, API interaction, and text handling. In addition to this, Python files are projects also include several directories, such as VectorStore, which stores all the vector database and related files. In VectorStore, we have image references and PPT references, which stores image extracted or converted PowerPoint slides and its related files. This structure allows us to efficiently manage different types of data and maintain a clean separation of concerns in our code. Before we dive deep into our React QnA chat application code, let's generate an API key for NIMS by going on to build.Nvidia.com. And we'll select Lama3 as our LLM model for the application. Here you get various options to try out the model or just boilerplate code to get this model or NIM, optimize NIM onto your application. But here we are trying to generate an API key. So just click generate and there you have it. Just copy this API key and you can use wherever you are setting the environment variable. But in my case, I'll set this on my terminal. Assign that API key to a variable named nvidia underscore API underscore key as an environment variable and you can do that with this command and press Enter. Let's start with utils.py which contains various utility function for our multimodal project. Here we are importing necessary libraries and setting up our NVIDIA API key. This is an alternative way from what we did in the terminal, but you can choose either one of those, whatever is better for you. And then now we load image processing function, which can handle image conversion to base 64 format, which is necessary for sending images to NVIDIA NIM APIs. Then we use image analysis function, which help us identify and process the graphs or charts in our images. We use NVIDIA's NIM APIs where the model are hosted to generate a description of these visual elements. And now we are using these functions to interact with NEM API to describe images and process graph which are crucial for our multimodal understanding capabilities. Next we'll create some text processing functions to process text from PDFs, allowing us to extract context around images and tables and group text blocks efficiently. And now we'll do some file handling, which handles saving uploaded files, which is important for processing user provided documents. So that covers the main utilities in our utils.py. These functions form the backbone of our image and text processing capabilities. Now let's move on to the document processor.py. Now let's go through the document processor.py file, which contains functions for processing various types of documents. So first we start by importing necessary utilities and notice that we also importing functions that we just defined in the utils.py. Now we'll create a function that is crucial for extracting content from the PDF files, including text, images, and tables. Now we write function for handling the extraction of table and images from the PDF files including processing them and storing them. Now let's work on the PowerPoint processing and here we are writing four different functions that work together to process PowerPoint files, converting them into PDFs and then to images while also extracting text and notes. And now write the main function, which handle the loading and processing of various file types, whether they are individual files or entire directories. So that covers the main component of our document processor dot py file. Now let's move on to the main script, which is app dot py, which includes all the GUI and function calling for all these scripts that we have just written. Now let's move on to app dot py, which is the main application file for our project. We start by importing necessary libraries and set up our streamlet page. Notice how we are using Lama index components and our custom modules. Then we set up our embedding models and language models using NVIDIA's name integration with Lama index. And then, uh, we create the index, which uses the vectors database using Milvus and stores all our document embeddings. Now this is the main application where we set up a streamlit interface. It provides option to upload or directory processing and sets up the chat interface. And now this section handles the chat interface, processes user queries and displays responses. So that covers the main component of our app.py files and all the other Python scripts. Now let's try running this code on terminal. Before we run the code, let's create a conda environment and install all the dependencies that we need for this project. First, we'll create a conda environment using conda create command. I'll name this environment as gpurag and I'll have Python 3.0 so it will create the environment. Now let's activate the environment that we just created by using conda activate and the environment name. Now we are into the environment. Let's install all the required dependencies by using pip install method and passing in the requirements.txt file which includes all the dependencies. Now dependencies are installed. Let's try running this app using Streamlit. To do that, you mainly use streamlit run and app.py. It should generate a port number where you can use to look at the web app. Here it has given us the local URL, network URL, and external URL as well. It's 8501.0.1. So let's try to go into the browser and see what's there. So this is how the UI looks like, where you have an option to upload files or enter a directory path where it will process all the documents present in that directory. So in this case, I'll upload mix of documents, which is a PDF of a blog from NVIDIA developer, which includes images as well as graph doing various performance comparison or the speed up. And then I'll also add an image or a slide about Tensorati. So let's try importing those files so once the documents are uploaded to the server i can process these files where it is generating embeddings and creating a vector database where you can query or ask questions about these documents that you just uploaded. So it may take some time or while based on the size of the documents. Now that the index is created, let's try asking questions based on the document we just uploaded. So first question I'll ask is like, what is the variant about Lama that is the variant about llama that is the fastest. So based on the chart in this blog, it answered correctly which is 7bq4. Now let's ask how does CUDA graph lead to substantial performance? performance. So again it used the context and then let's wrap it up by asking who's the author which is Alan Gray based on the doc we just uploaded. So there you have it. Users can now upload documents of various input types including text and vision, ask question and receive answer based on the content of those documents in real time. So in this tutorial, we have covered how to build a multimodal AI-based RAG application for enhanced understanding using GPU accelerated components from NVIDIA and open source tools. Feel free to experiment with the provided scripts and explore further. For more information and resources, check the links in the description below.
Building Multimodal AI RAG with LlamaIndex, NVIDIA NIM, and Milvus | LLM App Development
This video explains how to create a multimodal AI retrieval-augmented generation (RAG) application, including the following steps:

1.  Document processing: Convert documents into text form using vision language models, specifically NeVA 22B for general image understanding and DePlot for charts and plots. 

2.  Vector database: Explore the power of GPU-accelerated Milvus for efficient storage and retrieval of your embeddings.

3. Inference with Llama 3: Leverage the NVIDIA NIM API Llama 3 model to handle user queries and generate accurate responses.

4.  Orchestration with LlamaIndex: Integrate and manage all components seamlessly with LlamaIndex for a smooth Q&A experience.

Learn more with this notebook:  https://github.com/NVIDIA/GenerativeAIExamples/tree/main/community/multimodal-rag 

Join the NVIDIA Developer Program: https://nvda.ws/3OhiXfl 

Read and subscribe to the NVIDIA Technical Blog: https://nvda.ws/3XHae9F

#llm #llms #llama3 #llamaindex #nvidiaai #nvidianim #langchain  #milvus
NVIDIA NIM, code review, LangChain, llamaIndex, Llama 3, Milvus, NIM APIs, Mixtral, NeVA/DePlot