. Welcome to this tutorial on building a customized question and answering chatbot application in Cloud using open source and NVIDIA tools. Today, we will walk you through creating a powerful GPU accelerated QnA-based text assistant that can answer question based on your proprietary data. Let's take a quick look at what we will be building. As you can see our chatbot can quickly process and responds to questions about the data provided. Now let's try to break down on how we build this end-to-end using NVIDIA and open source tools. Before we dive into the specifics of our chatbot, it's important to understand that the ecosystem that we are working with. NVIDIA provides a full stack ecosystem for large language models that integrate seamlessly with popular open source tools. This ecosystem offers flexibility and performance, allowing developers to switch between different components as needed. NVIDIA not only provides proprietary tools, but also contributes significantly to the open source community. Tools like Nemo for building and training LLMs and Triton for inference servings are prime examples of NVIDIA's contributions to open source. These tools integrate well with broader AI ecosystem, providing developers with powerful GPU-accelerated options for building LLM applications. Nemo goes beyond just a training framework. It includes several open source components that enhance its capabilities for developers. Two of the notable Nemo components NEMO Data Curator, which helps simplify the process of data curation. It helps extract, deduplicate, and filter information from large amount of unstructured data, ensuring high quality and relevant datasets for training. The second component is NEMO Guardrails and this helps implement safety measures and control for model output. It also allows developers to add guardrails that prevents inappropriate responses, enhancing the reliability and safety of AI application. You can learn more about this by going on to github.com slash nvidia slash nemo. Now let's quickly understand about Retrieval Augmented Generation or RAG, the core technology behind it. RAG enhances the LLMs by giving them access to external and up-to-date knowledge. Here's a brief overview. We start by our enterprise data or proprietaryety data. This data is then processed into a vector database. And the way it works is you take the data or text and split them into chunks. And then those are converted into vector embeddings, which are then stored in a vector database for quick retrieval. So when a user submits a query at inference time, it is converted into an embedding, and then it is used to search through the vector database that was created using the enterprise data. The most relevant information is then retrieved and ranked. So both the query and retrieved information are then fed into the large language model as in context, which generates a response based on its pre-trained knowledge and the specific relevant information from our database. So this process allows the chatbot to provide responses that are both general and specific to our existing data or enterprise data combining retrieval and generation for more accurate and contextual answers. Now let's look how we implement this pipeline using NVIDIA tools and the other open source components. So let's look at our pipeline and first we have some paper from archive which contains some research about latest LLM technologies. This data about research paper fuels our chatbot knowledge base. We'll be using GPU accelerated tools for faster processing and more efficient handling of large dataset. Specifically, we'll use NVIDIA inference microservices or NIMS for hosted LLM and embedding model and Milvus for GPU accelerated vector search. For embedding model, we'll use NV embed, which leverages NVIDIA's GPU acceleration to quickly transform text into high dimensional vectors. For vector storage and similarity search, we'll use Milvus, a GPU accelerated vector database that can handle large-scale similarity search with impressive speed. It's important to note that using GPU indexing and querying requires numerous query vectors or exceptionally high request pressure for maximum throughput and reduce latency compared to CPU indexing. So the GPU acceleration in NV embed and Milvus significantly enhances the performance of our chatbot, allowing for faster indexing and querying for our archive papers. For actual question answering, we'll use a large language model, specifically LAMA3 7db instruct model, which will be accessed through NVIDIA's NIM API. This gives us the power of state-of-the-art language models with the speed of GPU acceleration, resulting in real-time responses and improving the user experience. Finally, we use LAMA index to orchestrate the entire process from query processing to retrieving relevant information and generating responses. We'll use Gradio to create a user-friendly interface for our chatbot. Now let's dive into the code. Before we dive deep into our React QnA chat application code, let's generate an API key for NEMS by going on to build.Nvidia.com. And we'll select Lama3 as our LLM model for the application. Here you get various options to try out the model or just boilerplate code to get this model or optimized NIMS onto your application. But here we are trying to generate an API key. So just click generate and there you have it. Just copy this API key and you can use wherever you are setting the environment variable. But in my case, I'll set this on my terminal. Assign that API key to a variable name,vidia-api-key as an environment variable and you can do that with this command and press enter. Now let's make sure we have all the necessary packages installed. So I've included all the packages in the requirement.txt file and you can install with the pip installation method, which is pip install hyphen r and requirements y. It will start installing all the dependencies before we start writing the code. Now this is done. Let's move on to our code editor. Now let's dive into the code. Before we start, let's import all the necessary libraries and configuration for our application. We are using DRADIO for the user interface, LAMA index for document processing, and querying and NVIDIA's embedding and language models through NIM API. Let's also check if NIM API key is set as environment variable or not. Then we'll set up some global variables to store our index and query engine. These will be initialized when the documents are loaded. Now let's create function to handle document loading. The get files from input function extracts the file names from the file objects, and the load document function reads the file, creates an index using a mills vector store and sets up a query engine. Next, we'll implement the chat functionality. The chat function handles basic question answering, whilst the stream response function enables the streaming of responses, providing a more interactive experience. Now let's set up a Gradio interface. We'll create a file input for document uploading and then button for loading documents, a chat interface for asking question and then a clear button to clear the chat history. Now we'll connect our interface component to their respective function. The load button will trigger the document loading. The message input will initialize the streaming response and the clear button will reset the chat history. Finally, we'll add the code to launch our GradU application when the script is run directly. Now let's move on to the terminal to run this app. To launch our app, simply to run this app. To launch our app, simply type python app.py on terminal and it will start the web server where you can access the app. So here you can see it is this web app is running on local URL, which is local host with the port number 7860. Let's try to go onto this URL. Now that we have our RAC Q&A chat application up and running, let's demonstrate how it works. First, we'll import some documents. For this demo, I have research papers from NVIDIA. To import these documents, I'll click anywhere in this area to select my files. As you can see, I can select multiple files at once and in this case I'll choose both the papers. With our file selected, I'll click on the Load Documents button. This triggers our document loading process which reads the file, processes their content, and creates an index for quick retrieval. Keep in mind that creating this index isn't necessary with every application load unless you aim to incorporate new data. Great, we can see the confirmation message that our documents have been successfully loaded. Now we are ready to start asking questions about the content of these papers. I'll now ask what's the difference between Dora and Laura. Great. And there you have it. We have created a powerful drag Q&A chat application which is hosted on Cloud with streaming capabilities. Users can now upload documents, ask question, and receive answer based on the content of those documents in real time. Feel free to experiment with these documents and questions to see how it performs. In this tutorial, we have covered how to build a customized Q&A chatbot hosted on cloud using GPU accelerated components from NVIDIA and open source tools. Feel free to experiment with the provided script and explore further. For more information and resources, check the link in the description below.
Building LLM Assistants with LlamaIndex, NVIDIA NIM, and Milvus | LLM App Development
In this video, we dive into the essentials of creating a Q&A chatbot. Here’s a quick overview of the process:

1.  Embedding Creation: Learn how to use NVIDIA NIM microservices  to transform your text into high-quality embeddings.

2.  Vector Database: Explore the power of GPU-accelerated Milvus for efficient storage and retrieval of your embeddings.

3. Inference with Llama3: Find out how to leverage the NIM API’s Llama3 model to handle user queries and generate accurate responses.

4. Orchestration with LlamaIndex: See how to integrate and manage all components seamlessly with LlamaIndex for a smooth Q&A experience.

Learn more today with this notebook:  https://nvda.ws/4dRMwQe 

Join the NVIDIA Developer Program: https://nvda.ws/3OhiXfl 

Read and subscribe to the NVIDIA Technical Blog: https://nvda.ws/3XHae9F





LlamaIndex, NVIDIA NIM, Code Review, Milvus, LLM Assistant