 Hello, my name is Elias Eki. I'm an ML specialist in Google and welcome to this world through where we're going to demo the end-to-end GNI AppStarter Path, a resource we created to accelerate the path to production for building GNI applications in Google Cloud. So let's start with it. Imagine you are in a scenario where like you are you build a collab, a notebook, and you build your own agent. Yeah, in this case, for example, I'm using Langraf and I built my own really complex Langraf agent with a Gentic rug to power our Vertex documentation. In this case, it's the system instruction documentation page. So then the agent works well, it's working well in my Colab, and I'm looking at how I need to professionalize it. So this is where the starter pack can help, essentially. So the idea is I can navigate to the Generative AI repository, and it's under Gemini, Sample Hubs, and to NGNI starter pack. There are also short links you can use to reach the same page. And then essentially the way to consume it is by downloading this zip file here and unzipping it. As part of this demo, just to reduce time for the demo, I already downloaded it, opened it up in my IDE, and I already created a GitHub repo so that we can use it to store the code. And I created a set of three empty GCP projects to deploy the full solution end to end. So the idea is really rich production in minutes here. So I'm going to open my ID and with the starter pack already downloaded. And let's have a look quickly at the architecture you see here. Hopefully you can see. This is a really high-level architecture, but hopefully it gives you the full idea of the end-to-end. First of all, we have an application in Cloud Run. This application in Cloud Run is hosting FastAPI, which is a framework to do low latency serving, and our agent. Our agent can be built with Lanchain, Vertex SDK, Lama Index, and so agent. Our agent can be built with land chain, vertex SDK, Lama index and so on. Coice is yours in the way in the SDK used to be the agent. All the code around the application is stored into this application folder you see here. We also have the infrastructure as a code and CICD. This is installed in the deployment folder, along with a good remit to explain the end-to-end process. We also have like a vector store and data storage analysis. Again, the Terraform stuff will build everything from scratch for you. And the chain or agent you use will use an LLM, and that LLM is defined within your chain or agent again. We also have a notebook folder with a getting started notebook, which hopefully will help you in developing your first agent or chain, should you not have already an agent or chain developed. So this is more like a way for you to start familiarizing with the starter pack. And at the bottom of it, we even showcase how to do evaluation. So this is more like a placeholder for evaluation, but really we showcase how much evaluation is important before you reach production. So in this case here, we can actually start testing the service locally. Yeah, so as I was mentioning, we use FastAPI and as part of the FastAPI application, essentially we offer different endpoints so that the client can reach those endpoints. One is to stream events. This is the endpoint used to power the chat application. And the other one is for feedback collection. The streaming events endpoint, if you see it here, will call a chain. So this chain here is nothing more than a really really really simple lunch in chain in this case here which is essentially a recipe bot planner so it's essentially saying you're a culinary assistant you need to provide recipes for the user yeah and so fast API is invoking this chain here we need to do point reinstall and install the right packages. Okay, now we can run the playground. Okay, so for example, this is like a fully fledged 3-minute application. Of course, it's for prototyping purposes, it's not production already, this part here. But you can do many things. For example, you can ask a question and it is a multi-modal so for example you can upload images so you can ask for example give me a recipe based on this image it supports any kind of modality you can for given conversation, you can provide feedback here and this feedback will go in the server, in the FastAPI server. So you can kind of replicate what the user will do in production. You can also save chats. This is pretty important when you want to build a set of ground truth conversation that went pretty well. You want to use it for evaluation or tuning purposes. And so, yeah, it's essentially a way for you to really test the service before it goes to production. But what I wanted to show you is that as part of the application, right now we are using the chain.py I showed you just now, which was the recipe assistant. We also offer a set of pre-built other kinds of application. One of them is a RAG application to showcase how you can create a RAG application with dummy data. And the other one is a dummy LandGraph agent. So what you can do is to really like just replace the chain you are importing and this will automatically, you will be able to test the other agents. So for example, if you want to use the LandGraph agent, just need to replace the importing line here. And then we go back to the playground, open a new chart and we can ask like, I watch the weather in New York. First time we'll take a little bit more. So you can see actually this because we are using a LandGraph agent, this is performing a tool call. So we use function calling in the scenes with LandGraph to power a tool call. And this is essentially being forwarded to the Streamlit UI and everything is streaming back. So essentially in this case here, we're capturing the event. So imagine you have a real application, you might want to give citations to the user and things like this. Of course, you cannot even like edit the message. Like you can say here like San some front and redo it again. So something similar to like the UI from Bart. Yeah. Okay, so, but in this case, we are just using dummy use cases, but I do have my own chain, right? So I might want to use that. Yeah. So how do I do it? I mean, it's pretty simple. Essentially, you will just need to, in this case here, the code is not in a great state, but the idea is that you will be able to copy paste this code here, go into, let's comment for one second, this line, go into chain.py and paste the code you develop in the notebook, yeah? So this is the land graph we used before. So like server.py, it's importing chain.py, which is still running. You can see the application is restarting from scratch. Yeah, and it's loading up. Okay, so for example, now we can go back to the playground. And if we ask a question like, what models do support system instructions in Vertex? It should be able to respond to it. So you can see it's actually performing one tool call. Yeah, nice. So essentially, this tool call you can see is actually powered by the documentation we provided there. So essentially everything was downloaded on the fly and showcased to the user. So we can say yes, very nice. Okay, perfect. So we can see the application is working locally, fine. Now, as a developer, there might be additional steps to build confidence in the application. Of course, you need to do evaluation. But let's say I already done the evaluation. I already have confidence in my application, and I want to deploy to production. So how do I do it? So let's go back here, and let's close for a second. Streamlit. So I'll need to go to the deployment folder. And now everything I'm going to do is actually pretty well documented, even with videos, into the Regni file to the deployment folder. And now everything I'm going to do is actually pretty well documented, even with videos, into the README file in the deployment folder. So everything I'm doing, you can replicate it yourself by just looking at the README. So we document how the CICD will work, and I'm going to explain it to you as well. So the first thing we'll need to do is to deploy all the infrastructure we need with Terraform. So this is required because we might need to set service accounts, IAM permissioning, artifact registries, enable the right APIs, enable things like log sync so that every login we do lands into BigQuery. So the way to do it is pretty simple. You will need to change this file here. So this file here will ask you for a project ID for production, a project ID for staging, a project ID to run the CICD. Then you will also need to provide a link to your own repo. In this case, we use Cloud Build to do all the CICD. So as part of the, if I show you here, as part of the CICD project, I already have a repository connected to it so that I'm accelerating the demo, but ideally you will need to do this step here and pointing Cloud Build to the right repository you want to use for the process. In this case, I'm using this repository here in GitHub. Okay, so just to accelerate the time, we are going to replace all the variables in one go. So we paste this line here. So you can see I'm using the project ID I just defined earlier, staging project ID I just defined earlier, the staging project ID I defined earlier, and the CICD one. Yeah. That's good. The only thing I need to do is to navigate to the deployment folder, terraform, run terraform init, initialize the environment. And then I will run terraform apply and pass the variable file we have there. So this is going to create 71 different resources for me. So it's a lot of stuff. So it will take 30 seconds roughly. Again, doing this infrastructure, if you want to do it manually, it will take you maybe days and it will be full of errors. So Terraform is a great way for you to accelerate this process and build and use code to define all the infrastructure you need. Okay, you can see cloud service accounts are being created, of course, like users will be able to take the Terraform code and extend it for their needs, right? Imagine a user might need a Cloud Armor or a user might need a load balancer. It's pretty easy to extend this Terraform code for their needs and their system. So you can see we created 71 resources. Perfect. Okay, so now everything is connected together. Cloud build will list them for everything that we do in our repository. So for example, if I'm a developer and I'm going to make a change in my chain, so in this case I have this agent, and I'm going to make some edits, like I'm going to write some edits here. And then I want to make a commit to Git. We can see we just changed the file. What I will typically do is that I will open up a pull request. So when we open up a pull request, we are going to make a commit and push it. Oh, lost. Nice. So we can actually see the pull request here in GitHub. And as you can see, GitHub will trigger Cloud Build. So as a developer, I will be able to know if Cloud Build is having a successful launching test or not. So what's happening in Cloud Build? We can actually see directly in the UI here. You can see that there is a new run happening in Cloud Build. And it is essentially running for me unit test and integration test. So we are checking that the application runs fine before we merge our code into the main branch. Now, let's accelerate the process. Once you deploy it, it will look like this. So unit test and integration test will be successful. And of course, you can run it also locally. But let's accelerate for a second and let's merge directly our PUT request. When we merge our code into the main branch, you will see another process being triggered by Cloud Build here. The pipeline is called CD pipeline. So this pipeline is a little bit more complex cause it will will have the responsibility to deploy our application towards production. So the steps we will do here are the following. First of all, we build a container containing the fast API server, all the right dependencies, and then we push this container to artifact registry. Then we take the container and we push it to Cloud Run. And then finally, we do load tests. So we test the application in the staging environment. Once the load tests are successful, we are able to trigger a production deployment where we ask a user to confirm or not the production deployment. So an example of a successful load test, sorry, an example of a successful CD pipeline will look like this. And as part of it, you can actually, as a user, before you reach production, you will be able to inspect what's happening in your application and see a report of, like for example, the load test of the application. So for example, we're using Locust in this case here, which is a framework to do load testing. And you can see that as a user before reaching production, I will be able to see what's the average latency of my application, what's minimum, maximum milliseconds. You will be able to see like total number of requests over time. And as many as you like total number of requests over time. And as you increase the number of requests, what's the performances of your application. Yeah. So these things can be super helpful before you reach production. Now, the application will run for like three minutes. So let's wait for it because I want to show you like the full end to end, but I want to also unveil a little bit of the magic behind it. Yeah. So essentially if I show you here, this graph here, the idea is like whenever a developer is triggering a pull request, we use Cloud Build and the Terraform code was instrumental to do it. We use Cloud Build to trigger trigger this pipeline you see here, which is in the deployment folder, CI PR checks, which is a really simple pipeline, which is installing the packages, running unit tests and running integration tests. And the tests are defined in the test folder. So we have unit tests, and we have a pretty comprehensive unit test layer, integration tests, same thing here. We test everything end-to-end to check that the service is running fine. And then finally, we also have the load test here, which everything you can run locally, by the way. So you can always test before you deploy. When it lands in main, we do the same process I showed you before for the CD part. So you can see now that the WorldCA CD was completed. And as a user, the last step I will need to do will be to approve or not the deployment to production. So I may say good to go and approve the build. This will only run the last cloud build pipeline, which is pretty simple, is just triggering the deployment to production. Yeah, so nothing really, nothing complex, let's say. So you can see this is deploying the container in Cloud Run for us. We can actually have a look at deploying the container in Cloud Run for us. We can actually have a look at what we use in Cloud Run. So in this case here, and of course, like Cloud Run here, it's the default, but you are free to diverge. In this case here, the deployment is pretty simple. We are just deploying Cloud Run and we are saying it needs to have only a minimum of one instance. So that the service is always up and running for the users. So we have a good latency. And we have a concurrency of 40 requests. So 40 users in parallel can send requests to one service. But of course the service will auto scale over time. So yeah, it's almost finishing. I can also, in the meantime, I can also show you what's happening on the observability side, because as part of the FastAPI server, we are logging every single request that goes to the application, yeah? And so like if I, oh yeah, it just deployed, so I can actually show you in real time. So if I copy the URL of the application we have in production, we can go back and trigger make playground. So in this case here, we have a remote URL. So instead of using localhost, I'm going to paste my remote URL, and I'm going to authenticate the request. So models supporting system instruction in vertex. We will see the production model responding to it. Yeah, perfect. So you can actually see like end-to-end kind of scenario the production model responding to it. Yeah, perfect. So you can actually see like end to end kind of scenario where we deploy the service to production and the production endpoint was able to respond with the right tool call. So the agent is finally deployed to production in around 30 minutes. We can give feedback again and things like this. And the thing I wanted to show you is what happens once we use the model from production. So once the model is in production, we can navigate to the production project. And in there, you will be able to see the Cloud Run endpoint up and running. And you will be able to see something pretty, let's say special, which is all this stuff is happening here. So essentially what's happening is that the model is giving us full transparency of every single step happening in our application. So if I go to Cloud Trace, like for example, we'll be able to see these dots you see here here which are essentially let's reload it, which are essentially the full trace of our agent. So you can see these are actually the reasoning step of our agent in the production endpoint. So you can see the agent first interacted with the task agent and method, then he started the retrieving documents, then he started grading documents, rewriting them and producing the response to the user. And for every single request, I'll be able to see input and output of what happened. Yeah. So I'll be able to see, you know, which document I use and then everything will be logged. Because cloud log in will expire in 30 days, everything is going to BigQuery. So everything is landing in BigQuery. And again, the Terraform here was pretty important to set up this in seconds. So this BigQuery table here will contain everything we are logging from the application. So if I run it, I should be able to see all the logs we just produced by me invoking the endpoint. Yeah. And also same thing for the feedback side. We have a table hosting the customer feedback. So also these will contain everything from that now the final part the final part is that this stuff doesn't make any sense for us right it's pretty ugly to see a table like this i want to have a dashboard so we are also offering a dashboard as a template to view what's happening in their application so for example in, in this case here, I'll be able to see that, what requests are going through the application, what's the feedback over time, how many conversation I have, average feedback score and things like this. And for every conversation, I will be able to see what the human and the AI are responding. So this concludes our world through. Please don't hesitate to reach out for any feedback or question you should have. There is an ideas here that you can use for that. We are actively looking at your feedback to improve the starter pack over time. So stay tuned.
E2E Gen AI App Starter Pack - walkthrough
Thanks for watching! You can reach out to e2e-gen-ai-app-starter-pack@google.com for any question or feedback.