 Hey, Prompt, what would you say is the open source edge of large language modeling today? Well, Dr. Gregg, I would probably say it's got to be Lama 3. Hmm. And Wiz, what would you say is the open source edge of language? What would you say is the open source edge of language? I'm probably going to say Gen Z slang kind of conversation. The young kids got us covered, right? And that new, new LLM. Well, what do you guys say we combine these two edges of language and language modeling today? I mean, we got to link up, vibe well, and make a dope team today on Llama 3 and Hugging Face. Right, boys? Sound like a plan? For real. Come in, Greg. Prompt Engineering, Wiz. Today, we build, ship, and share together. We'll see you boys back in just a minute. My name is Dr. Greg. That was the Wiz. And our new friend, yet to be revealed, Prompt Engineering is joining us today. You may know him from YouTube or his other work on open source projects like local GPT. Today, we talk about how to do end-to-end prototyping with Lama 3 based on how to actually understand Gen Z language. We'll walk you through everything you need from data to fine-tuning to deploying a scalable endpoint to putting a UI and serving it up so that others can use it publicly. We're going to discuss some of the intricacies along the way, and we'll have a lot of fun exploring Gen Z philosophy as well. If you have questions along the way, whether for us or for Prompt Engineering, please drop them in the Slido link, or alternatively, go ahead and smash the chat. We will be getting a little bit more interactive today as well as we try to build something truly scalable we want you guys to help stress test it so let's go ahead and jump right in to end to end prototyping with hugging face and llama 3 brought to you by ai makerspace and prompt engineering as i mentioned as we align our aim for the day we want to build ship and share something that's really end to end. We want to do it with a scalable endpoint. We want to understand what that means exactly and how we're sort of hitting this nice in-between space when we leverage Hugging Face directly to do this. We want to also take a look at fine-tuning Lama 3 and understand how, in general, we can kind of leverage these open-source models through endpoints directly through Hugging Face. We want to sort of put this in context as to, you know, the question everybody has, but why Hugging Face versus AWS or Azure or GCP? Shouldn't I be using one of those instead? It's a great question and it's worth some discussion today. So what we'll do is we'll walk through end-to-end prototyping, fine tuning. The focus is gonna be on this scalable end points piece today. We'll show you how to put a UI on it, deploy it, and finally we'll do some Q&A. So we're going to build an end-to-end prototype today. And we're going to try to see if we can't really build an AI that What's going on at the open source edge of language? Oof, for instance. Definition, used to express discomfort, surprise, dismay, or sympathy for someone else's pain. The sound oof has been used when a player dies in a video game since the early 2000s. Gen Z, oof has been used when a player dies in a video game since the early 2000s. Gen Z, oof. You've probably heard it if you've got a teenager at home. But how can we learn maybe a little bit deeper about how to leverage this type of language and how can we use AI to help us. Take Plato, for instance. Courage is knowing what not to fear. Say what? It's all about being aware of the things that could go down, but not letting that stop you from living your life. It's like, go down but not letting that stop you from living your life it's like you know what could happen but you're not gonna let that fear hold you back from chasing your dreams and living your best life you're gonna face it head on you're gonna come out on top that's what courage is all about, bro. Gen Z has so much to teach us. So much. And we can see how we can leverage AI to do this and to translate for us with our application today. Let's go ahead and prompt it with our friend, Prompt Engineering. Show us how it works, man. All right. This is what we're going to be building today. It's going to be a scalable front end, and let's prompt it. So I'm going to be using a quote from Aristotle. Excellence is never an accident. It's always the result of high intention, sincere efforts, and intelligent execution. It represents the wise choices of many alternatives. Choice, not chance, determines your destiny. So let's see what Gen Z is going to say about this. All right, so endpoint is running. I'm going to keep it real. Excellence ain't no coincidence, fam. It's always the outcome of high key intentions, grinding hard and making smart moves. It's a smart choice between many options. Not just luck that determines your path in life between many options, not just luck, that determines your path in life. All right, back to you, Greg. Oh, dropping the wisdom, fam. What I want to do is I want to ask everybody in the audience now, because as Prompt mentioned, this is a truly scalable endpoint. Isn't that right, Prompt? Yep, yep. Let's test it out let's test it out man let's test it out everybody if you can go ahead and we're gonna drop the end point and the space we're gonna drop the space into the chat go ahead and smash that end point thanks so much prompt we'll be back to you in just a little bit. And don't forget, smash that endpoint, everybody. Smash it. Let's see how many requests we can get this thing. See if you can overload it. I bet you can't. Write a script to see if you can overload it. We don't care. Let's see how much action we can get on this thing. In the meantime, let's talk a little bit about prototyping and put this all in context. When we prototype, we're generally going through a few stages. We're generally going through prompt engineering. How about that? Then we're often moving into RAG and doing RAG straight away before looking at fine tuning and ultimately trying to think about some sort of agentic reasoning behavior. And the way that looks is sort of along two axes is you can take prompt engineering and you can try to optimize what you're putting into the context window by doing RAG you can also take prompt engineering and sort of optimize the way the LLM is behaving the input and output you're getting from the LLM by doing fine tuning this is sort of moving the prompt into the model a little bit and kind of doing some rewiring based on many examples that you might put in a prompt in a one-shot two-shot few-shot sense generally you need both of these things potentially fine-tuning both an embedding model and a chat model for building more complex systems and of course you can use agentic reasoning for all of these things as well what we're going to do today in the interest of time and doing something that's truly end-to-end is we're going to just focus on fine-tuning. And the reason for that is because it's just a little bit too much to try to do RAG and fine-tuning and really focus in on endpoints today. So we're going to go through our process aligned with our ethos here at AI Makerspace of building, shipping, and sharing. Our build is going to be about curating data, in fact, creating from some of the data that we curated a data set that's going to serve our purposes. Then we're gonna train the latest model. We're going to ship, we're gonna actually deploy this thing to an end point. We're gonna serve it up so inference can be done on it at scale. And then we want this thing to be shareable to others, not just locally, but publicly. So we want to put a UI on it and make it public. As we think about building, what are we doing? Well, we want the data and the model to really play off of one another. And of course, the model doesn't really care what the data is, Of course, the model doesn't really care what the data is, but the data really does matter. We're going to use data sets and models on Hugging Face to push these things up to the hub. If you've been watching our channel so far, you've probably seen this before. We're going to give about 10 minutes to fine-tuning LLMA3 today to show you how this is done. When it comes to shipping, we're going to deploy our fine-tuned LLM model to an endpoint. We could also do this for embedding models if we were building, let's say, a RAG system, but not today in the interest of time, once again. But not today in the interest of time, once again. And as we think about sharing, the UI is oh so important. It's that chat GPT style chat interface that people want. And that you want to give your users and stakeholders. So there's a really easy way to do it. We're going to show you how to do a chainlet today. And also when it comes to deployment for a public URL, so easy to use Hugging Face Spaces to do this. One click deploy to a container and you're on your way. When we talk about kind of the LLM stack and all the things that are part of it, there's a lot of stuff on this slide. And the text is really small. part of it. There's a lot of stuff on this slide and the text is really small, but we can kind of walk through at a high level the fact that we need to do like data pre-processing and embedding. This is particularly important in RAG systems. We need to do prompt construction, retrieval, optimization, and that's not even to mention everything that's really kind of truly on the back end of the execution and inference and the serving up of the models that needs to happen. The reason I wanted to show you this diagram today is to show you just how simple the application is that we're making today. Of all of these pieces in the emerging LLM app stack, we're only going to use a couple of them. We're going to use open source models deployed directly through Hugging Face, actually via AWS SageMaker, but through Hugging Face nonetheless. We're going to use the OpenAI Python software development kit to actually glue this stuff together in the middle. We're going to use the OpenAI Python software development kit to actually glue this stuff together in the middle. We could use langchain, we could just straight build this thing out. A number of ways. The reason we're using the OpenAI SDK is because it works really well and it's pretty easy to implement and it's overall a good choice. pretty easy to implement and it's overall a good choice. And of course, Chainlit allows us to not only take input from the user, but as we saw, give output to the user. And with that, we're ready to get into the fine tuning piece. Remember, fine tuning is about moving from prompt engineering into a place where you have more examples. You're optimizing the LLM's behavior. You're telling the LLM how it should act. Now, there's three types of fine tuning that we generally are going to be leveraging. These are not mutually exclusive, but they all matter. Training the behavior for a particular kind of response. That's not really what we're doing today. Constraining the input output schema. We're not really doing that today. We're doing that kind of a little bit. We don't want it to be extremely verbose coming out, but most of all, we're doing a little bit of language training. We're training the model to interpret new words much better. And this language training is the focus of today. If you want to know more about the other types of fine tuning, check out the practical fine tuning event that we did not too long ago. When it comes to tactical, what we're doing, we're doing something called PEFT, quantized LoRa fine tuning, parameter efficient fine tuning using a quantized low rank adaptation approach. All that is to say, we're modifying that behavior by updating not all the parameters, but the parameters that we need to update in the attention layer based on the LoRa approach. And we're using a quantized representation of those parameters, those weights to do so. Now, if you want to know more about PEFT, you want to know more about LoRa, you want to know more about quantization, check out a couple of events that we recently did on PEFT, LoRa, and QLoRa. What we did today is we went and we took the Gen Z dataset, commonly used Gen Z slang, and we combined that with the power of GPT-4. So we gave it sort of a dictionary to provide ourselves with the sort of input-output pairs that were more of translations, actually, than question-answers in this case. And we used that to fine tune the LLM. Of course, the model that we fine tuned used these pairs and we leveraged off the shelf, Lama3, 8 billion, Instructuned because of course, Instruct tuned is always the move when you're pulling open source off the shelf. So with that, let's check out exactly how fine tuning was done, how we push data and model to the hub and how we use them. Wiz, show us how it's done, man. Oh, yeah. Okay. So as Greg said, basically we started with a list of data or terms, right? And then we turn that list of terms into this very basic training set using, you know, GBT 3.5 Turbo, sorry, GBT 4. And so we get things like, it seems like he's not interested in maintaining a serious relationship. Looks like he's not about cuffing for real. He just wants to keep it casual. This is the idea, right? We built a data set that we could use to fine tune our model. Then we fine tune our model. So this is the notebook that we use to do this. That's going to be shared with you guys in the chat. And the idea is it's pretty much what you're used to, right? So we're going to use transformers, TRL, as well as some supplementary libraries to train this. We're going to use this data set that we generated. You can see that it is posted on the Hugging Face hub. If for whatever reason you wanted a Gen Z English data set, feel free to go for it. So we load that up. We've got 105 rows, which is very little data. the English data set, you know, feel free to go for it. So we load that up. We've got 105 rows, which is very little data. And then we're going to use, you know, see that it's got the English sentence and it's got the Gen Z translation. Then we're going to create a prompt. The prompt is basically going to be, this is the Lama 3 instruction template prompt. The prompt is basically going to be, this is the Lama3 instruction template prompt. So we have our system message, which is contained in the system header. We're going to have the system message just be GenZify. And then we're going to end our system header and we're going to start our user header. And then we're going to pass in the natural language. And then we're going to close our user header. And we're going to start our assistant header. And then we're going to expect there to be some kind of translation. And so this is what we see. When we build this prompt, you'll notice we just, you know, take away the actual natural language. Same with the response. We create a create instruction helper function, which is going to turn rows of our data set into this, into the format we expect, which is this prompt. And we can see that in action here. When we create an instruction, we get our beginning of text start header for system, gen ZFI, start header for user. That was really funny. That's a natural language description, right? Or a natural language sentence. And then start header for assistant. I'm weak. End of text. There you go. Now we're going to load our model. Like Greg said, we're going to load this in four-bit quantization. So the idea is that we have this set up. And the basic idea, I'm just going to go ahead and make sure that this is shareable for you guys, and then I'll drop it in the comments. But the basic idea is what we do is log into Hugging Face. We need to do this because Llama 3 is a gated model. So what that means is that you can't use it if you don't basically ask meta. So you will have to log into this notebook and you will have to request access from Huggyface. There are other models available by certain, say, new research teams that will don't require the gating. We also have the ability to basically load this in 4-bit i'm not going to go into that here as greg said uh there you go and then we're going to set up our tokenizer we have to do this uh tokenizer swap because we're going to fine tune uh just part of the part of the thing so we we're going to add a padding token we're going to set as our end of sequence token, which is not typically something that we would want to do, but in this specific use case, because we're going to be using this padding technique later, we can get away with it. Again, we go into more detail in other events about that. We're then going to see, like, how does the base model perform, right? So we say things like they're in a very complicated romantic relationship, and the model says the drama. So they're in a complicated, huh? That's totally relatable. The gen-C-ification of relationships is all about the messy. And it's not quite what we want, right? So we can say, you know, things like she looks very attractive. And we get a response like you're saying she's got that gen-Z-ified glow going on, which might be gen-C language, but I feel like they're not that self-referential. So the model's like kind of on the line here. It's not doing a bad job. It's not doing a great job. Okay, okay, okay. But, you know, we want to make it better. So what are we going to do? Well, we're going to use LoRa. Again, go into that more in another event. And then we're going to finally do our fine tuning. Got a lot of text here that walks you through what the hyperparameters are doing, why we've chosen some of them. And the basic idea here is that we're going to train this thing for 10 epochs, which is going to probably overfit, right? So it's going to be pretty well overfit. But that's okay. You know, we're okay with that. This is just a translation task. We're going to do this low batch size. This is going to keep our GPU RAM pretty low. The reason it's so high is because we've actually loaded the model again, but you can actually get away with training this thing on a T4 instance, which is the free instance, which is pretty good. Then we're going to go ahead and we're going to set up our TRL trainer, which is our SFT trainer, Supervised Fine Tuning Trainer. We're going to finally train this model. 10 epochs, right? So we're overfitting for sure. We can see that our training loss gets very low. We're definitely overfitting, but it's fine. And then we can save this model. We're going to free up some memory so we can reload our model. This is going to help you be able to do this in the T4 instance. And then we're going to go ahead and we're going to push this to the hub at this address, right? So that we can load it on our inference endpoint later, which Prompt is going to take you through. The idea here is that we want to be able to load this model as an inference endpoint, so it has to exist on the hub in order to be able to do that, at least very quickly. You can certainly find more hoops to jump through, but the quick integration is available that way. And then of course, we're gonna go ahead and we're going to add that translation tokenizer as well. So if the tokenizer is not present in your hub space, it will not, the inference endpoint just won't work. Okay. Then we're going to load this into another pipeline, this time with our merged model, which is our fine-tuned model. And we're going to see how it went. We can say things like the greatest wealth is to live with little right and we get the response the most valuable flex is being low-key rich in your own mind much better than original right like definitely i'm not a gen z scholar of language right but this is this feels more succinct at least and it's doing what we want it to do and then we can say this this next one i was born not knowing and have had only a little time to change that here and there. And we get the classic. I was a noob and I've only had a hot sec to upgrade that. Absolutely fantastic. And that's all we have to do. So now that we fine tuned our model and we have that fine tuned model on the hub and we've kind of verified that at least for these cases, it works. And keep in mind that these are outside of the training set, right? We didn't train it on anything like this. These are just quotes from random philosophers and physicists. So, you know, it seems to be doing well. So we can move on to the next steps, for which I will pass you back to Greg. Yes, nailing it. Okay, learning so much here. Let's talk about the end points, okay? Now, just to contextualize, let's make sure that we're all clear on the language here. We want to be serving models so that we can do inference. All right. And the idea of Hugging Face Inference Endpoints is that it solves the problem that you need a GPU to run inference. And this makes it really easy to scale this out. Of course, you can host a single application on a single GPU on Hugging Face relatively easily. easily. What you can do with this inference endpoints is you can sort of have a little bit more control over exactly what's going on with the GPUs needed to run inference. And this idea of control is going to be a central theme of our discussion today. The way to think about the solution that Hugging Face Inference Endpoints provides is it's an easy way. I have a model. Now I have a model that's on Endpoints with a GPU. And I kind of control it. And so you can run this in production environments, like as an enterprise solution, it's possible. Now, should you? Well, it depends. Now, how's this happening? How is it happening? Well, it's happening because under the hood, we're using TGI, text generation inference from Hugging Face.. Now we don't have time to go into the math or details of TGI. If you want to see that at some point, let us know in the feedback and in the chat that we should do an event on it. But TGI is basically allowing us to do inference fast. So many tokens, really fast, basically allowing us to do inference fast. So many tokens, really fast. And there are lots of tricks that it employs to do this. Again, we're not going to talk to you about the details of these tricks for speeding up inference, but just know that it works dang well. I want to bring Prompt and the Wiz up to the stage here to ask like the question first, to ask like the question first, for a little discussion, Sash here, guys. A lot of people feel like if you're not using AWS, GCP, or Azure, that you're not actually building production LLM applications. First of all, is this true? What's your opinion prompt? Definitely not. There are a whole bunch of other options. And actually like if you're building on AWS or GCP, you will need to set up a whole infrastructure, right? But there are better options which will take care of the infrastructure part for you. Okay. Okay. So when you think of, you know, other options, what do you think are the reasons why people might choose something that's not AWS or GCP or Azure? So imagine you are a single person startup. Okay. You don't have a dedicated infrastructure team and you want to deploy LLMs at scale, right? So you can use these TGI inference from Hugging Face, for example, right? Hugging Face is one of them. In that case, Hugging Face is taking care of all the infrastructure for you. You just need to deploy an endpoint with the dedicated resources and that's it. Wiz, if you were a one person startup, would you use Hugging Face inference endpoints? I would use anything that makes my life easier, which does include inference endpoints. Yeah. The idea is when you're starting and you don't have a team and the had to then go and figure out how to either just use hugging face inference endpoints or i had to go like figure out how to use an entire cloud computing platform it's like oh man i'm taking hug and face inference endpoints all day now if i had the wiz or prompt engineering on my team, I might consider doing something a little more sophisticated. And this sophistication, I think is the next key point guys. I think this idea of control that we brought up earlier is very important here. I wonder if you can comment on this prompt. You know, my understanding, and maybe you can correct me or verify this, is that if you use a cloud computing platform, you have lots and lots and lots of control, like let's say all the control. But if you use something that is a little bit more managed, that is a little bit easier, that is a little bit less things you can change, like hugging face inference endpoints, that you're sort of just making this trade-off. Is that right? Is that the right way to think about it, Prompt? Yeah, I totally agree. I totally agree, right? Like in the beginning, if you're just applying something, you don't really want to worry about a lot of optimization. Once you have something up and running, yeah, you definitely want to move to something where you have a lot more control, do all the optimization that you need, right? So that makes sense. Yeah. Yeah. And Wiz, if you were going to optimize something after you had it kind of deployed, what would you be looking to even optimize? I don't even exactly understand this from a non-scale engineering perspective. There's a lot of costs that you incur by using these kinds of managed services, especially kind of these ones that are intended to be more simple. You know, it's always on, it's always available. The actual replica scaling is not that great. You don't have any real decision-making power over how many instances you have or how many you need. I would say, ultimately, you're missing a lot of that fine-tuned control that can really help you optimize cost. So it's going to be high availability. You're going to be able to make sure that it stays up most of the time. So you'll get all of that greatness, but it's going to cost a lot. You're not going to have a lot of fine grain control over reducing that cost as you begin to learn the habits of your customers. And as you begin to use, learn the habits of your business, you're just, you're going to be missing out. Okay. So the idea is you'd want to get something up and running. You'd want to take a look at it. You'd want to then assess like what's going on with this data, see how people are using it. And then as you optimize, you're really optimizing to serve up the model at a speed and at a cost to you that makes sense for your business. That's really what we're talking about at the end of the day, right? Yeah. The idea is like you have more levers, right? If you have more levers, you can get closer to your desired outcome than if you have fewer levers, right? So it's, yeah. Yeah. I think an analogy would be look at no-code tools versus writing your own code. In no-code tools, you are limited, but it's a lot easier to use in the beginning. And if you write your own code, that will give you a lot more control. Yeah. You know, Pram, if somebody's starting out there i mean would you generally recommend that people start with no code low code tools from the jump or would you you like, would it be obvious to people out there? Like, if they were going to use high code tools, that that's a good idea. I think a lot of people have this sort of fear of like, I don't know how to code, I can't build it, or I don't know how to code enough. So I need to keep learning how to code more before I can actually build the thing like what's your what's your advice to folks that you know don't really want to pick up a no code tool but you know potentially might be able to benefit and do things a little bit faster even if it's a little more expensive and not as not as cool necessarily in the meantime. Yeah, like the way I think about it is if you're not a coder and you need or you want to ship fast, right? It's actually really good to adopt these no-code tools in the beginning, right? You ship a product or idea and you validate it, right? And then comes the second part of optimizing it, right? So for that, you definitely want to, let's say, either learn yourself or hire engineers who can actually do it for you. Yeah, yeah. Yeah. And I guess that's sort of the way that we'll kind of wrap this little discussion up is that's kind of the best way from talking to you guys that I've come to think about this is we're almost kind of hiring Hugging Face as a partial engineer for us a little bit here. And we're sort of saying, we're going to give you a little bit extra money, not as much money as I would need to spend necessarily to hire an engineer to manage AWS for me, but I'm going to kind of hire you as a bit of an extension of my team to manage this a little bit more for me. But I am still going to be able to scale this out if I can find that product market fit, really with the enterprise solution as much as I want. And that's hugging face in print in front of 10 points in a nutshell, as far as I understand it. Do you have anything to add to that whiz? Absolutely not. I think that's, you guys covered it so well. At the beginning, right, you have money and no time. And at the end, you have money and time, hopefully, if you're doing it right. And when you don't have time, no low code is a multiplier. So, yeah. Awesome. Awesome. Well, we're going end to end today so let's get right back into the end to end action and before we go to our next piece of this we're going to see how we can ship now that we've built and for that we go to prompt engineering shows how to set this up now man all right so assuming everything goes well with the training and you are able to push your model to help so you can see something like this so this is the second version of the model that was trained. So I'm going to show you how to do the deployment. So this is a public model, so you can actually access it. And in order to deploy it, there are two ways. Either you go here and select Infos Endpoints. This will basically take you to a new endpoint that we where you can create click on uh new that will start the process of creating a new endpoint right so we actually need to look for uh that model that was trained you can look it up here or if you have access to the model that you just uh deployed you can click here on deploy, click on inference endpoints, right, and it will automatically populate the model repo for you. Okay, next, you need to provide the endpoint name. You can call it whatever you want. I will just stick to the default name that HuggingFace selected for me. All right, now this is a big model, you can call it whatever you want i will just stick to the default name that hugging face selected for me all right now this is a big model so we will need a powerful gpu to run this the good thing about uh the inference endpoint from hugging faces it actually gives you a suggestion or recommendation uh which gpu to use depending on the model that you selected and there are multiple providers so we're going to stick to aws we'll select gpu and we're going to go with the a10 gpu uh there's an option of automatic scale to zero so let's say uh there is no traffic on uh your endpoint for a while so you can actually select it uh to go to or go to basically sleep right so if I say after 15 minutes with no activity then it's going to go to sleep that will save you some money then you can also include options like protected so in this case in order to access this your user will need a hugging face token or you can make it public right we'll just keep it protected now you want to also look at some other options for example if you do auto scaling then you need to provide the maximum number of uh replicas that you want and the uh minimum number so the minimum minimum is one and depending on your needs you can auto scale it to whatever you want the task is going to be text generation we're going to keep everything else uh the same the only uh thing that we're going to be updating is the quantization so we're going to use bits and bytes to run a quantized version of the model rather than the full precision model and that will help us uh basically increase the entrance speed as well okay so once you go through all these options just make sure to select like the task is text generation right and select the appropriate quantization we did use the lot of doctors so that's why we're using the bits and bytes for quantization and uh you just hit create and this will start the endpoint creation process now this is going to take a few minutes so just bear with it and everything if goes well you will be provided with endpoint url here that you can use in your code so let me show you how you can do that within Python. Okay, so here is a notebook that we're going to be using. So first you need to provide your Hugging Face token because we created that as a protected endpoint. Okay the this is the url that you will get once the deployment is complete we are going to be doing async calls right so we have this very long text that is going to go as a query or prompt okay Then this function is the one that is actually making the API calls. So you provide your API URL along with the query that you're going to be passing on. Then there is this helpful function, which is going to make 500 calls to the same API endpoint. So we really want to stress test this and show you that it's actually able to make a lot of calls to this scalable endpoint. And it will hold. So that's why we're using this, right. And we wrap around everything within this async main function. So when we run this, it will start creating or start making a whole bunch of calls to the endpoint and we're going to get responses. All right. So this is how you quickly deploy and test the model. I will pass it on to the Wiz. Yes. Okay. So now that we've done this, let's take a peek at something that is pretty cool. So we can actually see our dashboard here. Now, if you notice this dashboard, right, you'll see that there's, there's been a lot of activity, you know, since we've started the endpoint over the last 30 minutes. So you can see here that we have the testing that we did where our requests was getting very high. And this is basically just showing the model chugging through all those requests. The basic idea is, you know, I'll extend this back out to the last three hours now so we can see the spikes. You can see the different times we test it. You can see that there is no, You can see the different times we test it. You can see that there is no, there's not a lot of incidences of 400 requests. It looks like we did have a few. Maybe someone was trying to programmatically access the endpoint without the API key. But the idea is that we're servicing a lot of requests without crashing, right? And we're not seeing any 500s, which is awesome. The downside, since we're not scaling, so we're only using one replica here, and you can see here that when we look in our configuration and we look at the, or sorry, when we look at our settings, we can see that we're not doing any auto scaling, right? So when we look at our settings, we can see that we're not, we're not doing any auto scaling, right? So when we have loads of requests, so if I just go back to that two hour window, right, we get these, these spikes in latency, right? So this is how that, that auto scaling can help us out because it can help us to better ensure that we have fewer spikes when we start sustaining a lot of traffic. There's going to be some lag time, of course, with the auto scaling, but it's not so bad. And you can see that like the median request latency is not so bad. Now, in the front end that we are using, you might see that it feels slow. That's because we're not using the streaming use case. And so it waits for the prompt to be fully formed before sending it back but the time to first token is not very long you can also see that under load we're using a lot of our card uh and uh you know very little of the rest of our machine but this is the idea right we can hammer this thing with those 500 requests at a time and it chugs along all right. So that's the idea. And that's how we can track the usage for us. We can also see our specific usage, how much time it's been up, you know, what our rate is and how much cost we've incurred. So for this, since we opened it before this event, it's been about two dollars and by the end of the event it will have been two dollars uh that we've spent to do this so not so bad uh but with this i'm going to pass you guys back to greg who will uh take us to qa but before i do i'll remember to uh say don't forget to like comment subscribe, subscribe, and ring the bell notification. We go live every Wednesday. We love to see you out here, and we'll switch you over now. Yes, thank you, Wiz. That was awesome to see. And we are actually going to do the final leg of our end-to-end prototyping next which constitutes the ui show you a little bit of what's going on behind the scenes with chainlit and for those of you that are wondering well chain, Chainlit or Streamlit, it's like, those are stupid UIs. Those aren't scalable, right? Well, I wouldn't be so sure. I was talking with Dan, the co-founder and CEO of Chainlit, not that long ago. And he told me that we had someone deploy an app that had 1 million users per week, Chainlit. So get some, Chainlit. That's pretty cool. And I certainly have heard, I don't have any verified quotes, but I've heard that Streamlit doesn't do too bad as you scale up either. It's like, really, what scale are you servicing? And when does this break? In the meantime, why not use it as a sort of member of your team instead of hiring a UI expert and designer? It's also got a really simple syntax that you can start the chat, you can keep the chat going, and that's pretty cool. Once we have a chainlet front end, we can deploy it to Hugging Face's spaces. We've talked about this before. We can one-click deploy. You don't have to have a GPU to deploy a space, but you can go ahead and attach GPUs super easily and even via hugging face inference endpoints to close us out we're going to go back to prompt engineering to show us the sharing part the oh so important sharing part of end-to-end prototyping as we get our work in front of others who are non-technical prompt over to you man okay all right so we're going to talk about how to quickly create this front end. So as Greg was saying, Chainlit is pretty amazing. It's very simple, straightforward. So here, you first need to provide your prompt templates exactly the same what we used during the fine-tuning process. In this case, you want to make sure that you stick to the LamaTree's prompt template so that it actually generates good responses. Then you're going to provide your API URL. So the same thing that I showed you before, right? Some headers for authorization, okay? Then pass on our query. This is the prompt template, which the user input is going to come in. That is going to be formatted. And we'll create a query based on that. And we will create a simple inbox where we'll get the input from the user and pass it on to our API to generate a response. Okay. So this is how it's going to look like in terms of the final UI that is deployed on a Hugging Face basis. And we can test this out. So let's say we're going to test it out again. Now you just want to make sure that this is running. We can pass this. All right. This would take some time, but we do get a response. All right. So this is how you deploy it. Back to you, Greg. Awesome. Yeah. Easy, peasy, lemon squeezy, quick deploy. Awesome to see that. As we sort of think about and reflect on the session here a little bit, we can say, okay, what did we learn? Well, we got pretty existential today and we learned that the greatest wealth is to live content with little. In other words, the most valuable flex is being low-key rich in your own mind. We also learned that being born not knowing and having little time to change is the nature of our existence. In other words, we're all noobs and we only have a hot sec to upgrade that. We only had a hot sec to upgrade our end-to-end prototyping game today, and we did it through the lens of building, shipping, and sharing. We hope you enjoyed each piece. Of course, we've got lots of content that digs deeper into all of these things. The star of the show today was API endpoints that solved the problem of running inference on GPU. Cloud computing not required, but you can use it. And this gets us right into Q&A. We've got a few questions in this regard. I welcome Prompt and Wiz back up to the stage here. And I want to sort of focus in on that piece of it first here. Aren't there inference endpoints in AWS through Bedrock? Wiz, like, duh, why didn't we just use that? I mean, there sure is. Yeah, there is inference endpoints through everybody. You know, and in fact, if you look very closely, the inference endpoints that we're using today are secretly going through AWS anyway. You know, it's just click the Hugging Face UI to get them. The idea is pretty straightforward. Like the Hugging Face ecosystem is set up to hold your model, your data, and it can also serve as the inference solution for that model. So that's why we went with Hugging Face. Now, you can, of course, use AWS, Azure, whatever your favorite cloud provider is, if you want to. But we wanted to show a simple, few-click way to get the same thing without leaving the ecosystem you're training your model in. Okay, all right. Keep questions coming, everybody. Prompt taking this one over to you. You've done a lot of fine-tuning on your channel Can you share tips about how to decide the amount of data for effective fine-tuning? of any given LLM Okay, this is actually a very controversial question So some of the research that I've seen is that you could um so so the some of the research that i've seen is like you could get a really good fine tune out of the smaller models up to thousand examples but those thousand examples have to be extremely good right you can't just uh like or the traditional terms were like garbage and garbage outright so if you have a million examples of bad data uh it's not going to learn. But if you have some good examples, up to, I think, a thousand examples, they seem to work pretty good, at least for the smaller models. And I think, gosh, what were you saying the other day? You were talking about fine tuning to us and you were talking about how the one point, the one in 3 billion models, they even sometimes they're suffering if you do too much quantization on their performance too. Isn't that, you said something along those lines. Yeah. Actually, like there's a couple of things. That's a good point. When you fine tune a model, a lot of people use quantized version of those because that gives you better inference speed and memory usage is pretty small, right? Now, the problem is if you have a smaller model, like, for example, 5.3, that is going to suffer a lot more if you quantize it heavy. So if you use like eight bit quantization versus four bit quantization, you would see a drastic change in the performance. Same whole tools for you in the bigger models as well. If you quantize bigger model, like a 16 bit, eight bit, four bit, you will see a difference in performance, but for the smaller models, it's a lot more prominent. Fine tuning legend over here, Prompt. Yeah, follow at Prompt Engineering on YouTube for more fine tuning tips. Wiz, to you on this one from Jorge, how much do you expect this setup's inference to cost? We saw it costed $2. Can we talk about this on a per token basis how should we be thinking about the cost model here that we're using yeah i mean for how you face it for some points it's easy uh if the model is going to be up for four hours it's going to cost you four times the hourly cost if it's uh going to be up 24 hours a day do the math if you want to think about it in per token costs you can think about it by uh thinking about how many tokens you expect to have through that period of time but it's it's a hourly or per unit time cost so uh in in this case i would i would more focus on that uh when it comes to like the replica and the auto scaling that is where it's going to be a little bit more complicated. In that case, I would think about doing the math to find it per token cost. And that's going to help you decide. Let's say you receive enough requests that you're going to have four replicas. Well, that's going to cost you 4x. How much time should you sustain that for? Yada, yada. So it gets more complicated into there. For this specific one though, it's going to be a dollar per hour, regardless of how many requests you get. And is that per hourly rate, is that something that you would look at as you went to cost optimize as you started to grow? Is that one of the places you would start? Oh yeah. If we know, for instance, let's say we need more than one replica at a time to service our requests with decent latency, right? Then we want to find times during the day where our users are really slamming the endpoint and then spin up more instances then and spin them down when they're not. So there's a there's a lot of different ways you can handle this in order to keep a cap on cost at the at the cost of some latency for your users nice okay okay um uh fee slides are available we will drop them on youtube in the comments as usual and then prompt i you, we got this question in the chat here from Edgar. It seems like it just will not go away. What do you think about the value of doing a prototyping event within a cloud provider? Why do we avoid that today, in your opinion? Well, I think it's a lot more complicated it will not be able to cover those in an hour right uh to begin with and i think for beginners it's really good to have something up and running uh and then you you can think about uh these cloud providers like for a lot of companies for example right uh they don't really have even an option of which cloud provider to choose. They will stick to the existing cloud providers. So even let's say for companies on GCP, they're going to stick to GCP, even if AWS is offering better prices, because usually they have long-term agreements in there. So to begin with, I would say just use something like this which is pretty easy to set up but once you pass a certain point where you want to look into optimization and stuff like that, then yeah, you can explore all these things and there are some really good tutorials out there. I want to kind of double click in on this as we end. Thank you everybody for the questions and we'll do our best to get the ones that are relevant to fine-tuning over to Prompt to answer async and sort of make sure that we get answers to some of the more interesting questions here. But one of the things Wiz and I talk about Prompt a lot is like the actual job of an AI engineer and where it stops and what it's not. And I think a lot of times when folks are asking for the cloud computing platform content, it seems that they're asking for the content that's going to be stuff that they're not typically doing in their day job as an AI engineer. Correct me if I'm wrong on this contextualization, Wiz, but this is the core idea that we're pushing. What are your thoughts on this? Where does the AI engineer job description and job duties, where does it run out? Is it at the cloud computing platform? Is it well within the cloud computing platform? How much cloud computing platform do people need to really know? So I think about this question, like, let's say a couple of years ago, people would think about data scientists. Okay, so data scientist role was really dependent on which company you're working at. If you're working at a startup, a data scientist would be like an end to end engineer who would be like doing the data cleaning, data extraction, model training, then deployment, right? And if you are working at a big corporation, then data scientist is more of like, okay, I have to create models. So for AI engineer, I think we are going through a very similar phase, right? It's really good to know different, I think, technology stacks. It's good to know the deployment part as well, but depending on which organization you are working in, I think your role is going to look very different. Right? Yeah, so that that will be my answer. Like I know it's not a clear cut answer. But yeah, it's a hard question. Yeah, right. It is it is. And lots of people want to know the answer. And you know, and basically, I guess, you know, it's like, there's a reason why we don't teach, you know, AWS, or GCP, or Azure, because again, we don't teach you know AWS or GCP or Azure because again you don't get to pick it and if you want to learn it you know where to go you know you know where to go go ahead and just learn away on AWS on GCP on Azure with their materials they are the experts and their people that have full-time jobs dedicated to making you understand that ecosystem just to add to this like I think it's really important to know the concepts. Like for example, we covered deployment today, right? So if you know how to deploy a model or like what it takes to deploy a model, then what platform you choose is kind of irrelevant, right? You actually know, need to know the steps that you need to follow, right? And as you said, there are a lot of resources available depending on which platform to choose. Yeah, we just, we got to drive the car. We don't always have to build the car. As LEGO Block AI engineers. Well, thanks Prompt. Thanks Wiz. And that is the bookend of our end to end prototyping applications event today with Llama3 and Hugging Face special guests, Prompt Engineering. Don't forget to like and sub, ring that bell, baby. And if you enjoyed this, you want to keep building, shipping, and sharing with us, join the AIM community. We would love to have you. There's folks talking about projects that they're building, shipping, and sharing all the time. It's really exciting to see what's starting to take hold, led by community for community. Join us in there. And you can start learning for free with us. In addition to our YouTube, we've collected a cohort on LLM Ops that we taught in 2023. We're looking forward to sharing more cohort-based courses in an open-source way in the coming months. And if you are ready to accelerate right up to the edge, we always have our AI engineering bootcamp that's ongoing. It's a seven-week accelerated learning experience from your first LLM application to your first RAG, first agent, first fine-tuning, laying chain, Lama index, assessment, production, and demo day. We've got a demo day coming up. We'll invite all of you guys to join live and definitely check that out if you're interested. We've got lots of other cool stuff coming down the pipe, but for now we'll see you next Wednesday live on YouTube. In the meantime, keep building, shipping, and sharing, and we will most certainly, as well as Prompt Engineering, do the same. Thanks so much, everybody. See you next week. Bye.
End-to-end Prototyping with Llama 3
Join Dr. Greg, The Wiz, and Prompt Engineering for an exclusive YouTube event!
Dive into the complete journey of building, shipping, and sharing AI applications with the Hugging Face Hub. Learn how to curate datasets, fine-tune models, and deploy them with robust API endpoints. Discover how to enhance your AI projects with user-friendly interfaces and finalize them for production using Docker containers. Whether you're new to AI or looking to sharpen your skills, this hands-on session will equip you with the knowledge to streamline your workflow and bring your AI solutions to life. Don't miss out—click to join us and transform your AI concepts into real-world applications!

Event page: https://lu.ma/llmappshf

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/3VAruCSULjSYF2UT7BMoof

Speakers: 
​Prompt Engineering
https://www.youtube.com/@engineerprompt

Dr. Greg, Co-Founder & CEO
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO
https://www.linkedin.com/in/csalexiuk/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/xm9MsAHV7oJsTxvT7