 Whiz! SDG, synthetic data generation. This topic seems rather large after having dived into it a little bit with you. What's really at the core of SDG? What is its essence? Well, Greg, it's going to shock and surprise you but it's generating data synthetically interesting so anytime we generate data synthetically we're making a call to an llm right we're we're doing some prompting to that lm that does what it look that is what it looks like, sorry. Okay, okay, okay. So is it fair to say that in the end, it's kind of prompt engineering all the way down this SDG thing? Yeah, yeah. The rabbit hole is deep on this one for sure, for sure. Yeah, as we see every time we dig deep enough into AI engineering, prompting is there. It's pervasive. It's always a part of it. It's a matter of how we're prompting, why we're prompting, what we're prompting for, how it can be useful to us. These are the discussions we're going to get into today with SDG. Wiz, you ready to try to cover this big topic? I'm pretty ready. It's going to be pretty awesome. Let's see if we can pull this off today. SDG, everybody. Welcome to YouTube Live Wednesdays with AI Makerspace. I am Dr. Greg. That's the whiz. Today we're covering synthetic data generation, an emerging technique that really is getting kind of a lot of steam in the industry right now. People are creating new tools, new techniques for ways to do this for specific use cases. We're gonna try to give you a bit of a holistic view before we dive in to a very specific use case using Langchain and Lang Smith, as well as a special test data set generation tool from the RAG assessment guys. And, you know, it's going to be kind of a cool topic to scratch the surface of today. We believe there's more in store as we continue the journey. So let's get into SDG for enterprise today. All right. This is also in partnership with Langchain. Shout out to the folks at Langchain for the cross promotion and support. As we align our aim to the day today, we want to make sure that we kind of get this broad overview of SDG. And we want to have some discussions. We wanted to sort of make this specific to a particular domain. You'll see why that's important as we get into some of the overview of SDG. But today we're going to cover kind of some financial services things we're hearing about from our partners, we're seeing out in industry, and we'll build a financial services application that'll give you a feel for how you might use this in your business. And then we'll also take a look at how to use Langchain and Langsmith on top of a Langchain application for doing evaluation, no matter how you go about generating that specific test data. We're going to show you a pretty cool way that uses an evolutionary approach today to do just that. So let's kind of talk first about what is the deal with this synthetic data? Why are people kind of sleeping on it? Why is it gaining a lot of popularity now? And really, what can we break this down into? We'll talk about how we can do evaluation with Langsmith, and we can evolve test sets to test our applications like the one that we'll build today, a question answering machine for financial documents. All right. So, you know, if we start kind of super meta here, we might ask about synthetic data, the same questions that we ask about data, namely, like, that we ask about data, namely, like, where do we even need data when we're doing AI, when we're doing machine learning? And the answer is like everywhere, right? We need data for almost all of the things, which is not helpful, really. Maybe a more useful question to ask is, how can I make my app better? And this is one of the things, if we root ourselves in best practices of prototyping, that we can start to answer and sink our teeth into. For instance, we're often starting our applications out by prompting, and we're getting as far as we can. When we're ready to sort of add more data or take it to the next level in one dimension or more, we might start looking at ways we can optimize, say, what we put in the context window by using a retrieval augmented generation approach or by, quote, giving the LLM access to new knowledge, more data. Now, because RAG is kind of rooting us in truth, is kind of rooting us in truth, maybe generating fake data to put into your indexes, not so much of a best practice approach for synthetic test data generation or SDG in general. But on the other side, if we go and we try to add data in a different way, for instance, optimize the actual performance of the LLM, the input and output, the task specific nature of what we're using it for. This is sort of, we can think about what small language models are trying to do today. This fine tuning is really teaching the LLM how to act or how to behave. Now, this is a particularly quite useful way that we might start leveraging synthetic data by providing more examples, by providing more high-quality examples. Maybe we can get a better fine tune. Oftentimes we're using both of these dimensions of adding more data to our application to increase its performance. But as I briefly mentioned, it's not just more data, right? It's more high quality data. quality data. This is the key. So any time where you can reasonably say, all right, I need potentially more data and I can create very high quality data here using a synthetic approach, that might be one that you might think about leveraging SDG. So one area that you might just sort of zoom in on here is anytime you have task-specific behavior, this could be a good use case. For instance, you want to get better at generating summaries, you want your app to be better at this. How about you generate a bunch of good summaries that are slightly different, right? Nothing wrong with that. That seems like a pretty useful way to start leveraging this. And we can think about task specificity down the entire spectrum. As we move from prompt engineering to fine tuning, and we're optimizing that behavior, we're going to be giving zero instruction only into one, two, three, four, many examples of the behavior that we want to see. This is the zero shot, one shot, two shot approach to prompting. And as we move down the task-specific spectrum, as we move from one to two to many examples, we get to the point where we have a thousand, ten thousand examples, and now we're in a fine-tuning paradigm. So this idea of task specificity exists along the spectrum, and we're simply moving from putting it directly in the context window to now fine-tuning it into the LLM. Interestingly, we've been looking at a few tools recently. For instance, DSPY is one of these tools, and we saw that really at the core of DSPY, this optimization that's been going on is really this bootstrapped few shot approach, meaning that we're sort of creating synthetically few shot examples few shot examples that will allow us to test out and further optimize our application. Whether we use it for simple prompting and optimizing, let's say, a closed source model, or for RAG, as we've seen in another event, we're using this bootstrapped few shot approach. This bootstrapped few shot approach is exactly synthetic data generation. It is a specific type and way that we can leverage it. And anytime that we're doing a bootstrapping approach that actually is adding value, approach that actually is adding value, that's really where we want to think about SDG coming into play in a task-specific way. All right. But that's not the only way, and we'll put a pin in this bootstrapping, that we can potentially leverage SDG for task-specific behavior. We'll come back to other ways that these patterns and these statistical methods overlap. But from task-specific information to domain adaptation, there's a lot more ways we can also leverage SDG. For instance, anytime you have jargon, medical, legal, financial terms, right? You can leverage tools like domain adapted pre-training, domain adapting of your language models. So this is the small language models. This is the SLMs stuff. Or you can also domain adapt your retrieval by fine-tuning your embedding model in a RAG system, for instance. All of these could potentially benefit from synthetic data generation. could potentially benefit from synthetic data generation. And in fact, when you use some of the best practice approaches, it's encouraged that you leverage synthetic data generation to do so. So we might call that SDG for domain-specific data. But what other kinds of data do we use? Well, we're always doing alignment, right? So we can, if we want to like get the fine, fine tuning knobs going on our application, and we want to do some alignment, we need some input output pairs. Oftentimes, there are great data sets out there you can pull off the shelf but if you have a very specific type of alignment and application and use case and customer you can leverage your very own data to bootstrap up new similar examples to try to improve your alignment by using sdg for alignment data and today the focus isn't going to be on domain adaptation or alignment or even fine-tuning in sort of a task-specific spectrum way. and using that evaluation as a basis for improvement of your applications each iteration that you move forward and you continue your production LLM application development. And this is really helping you ask, you know, is my app actually good at this thing I want it to do? And so you can use SDG for evaluation data. want it to do. And so you can use SDG for evaluation data. Again, all of these ways that we use SDG could potentially have humans handcrafting and curating specific pieces of data for specific things. Oftentimes, it's not exactly feasible or practical to do that. And in many of these cases, it really wouldn't produce much of a performance gain. So I want to bring up Wiz here to have a little discussion here just on kind of the magnitude of this SDG thing. So it seems like since it's synthetic data generation and we live in this data-centric ML space, and everything is focused around data. We really can use it at every single piece and layer of the stack. Isn't that right? We can basically use it anywhere. Yeah. I mean, just to be tongue-in-cheek about it, if there's data, we can use synthetic data instead of real data, right? That's the kind of idea. So every part of the stack that has some kind of data, we can substitute our own created data instead. And so when we think about using, quote unquote, fake data, it seems very dangerous to do that in general general if we just sort of state it that way. But I guess the purpose of today's event is to really find a few places where using that sort of not human generated data actually does accelerate our ability to put prototype applications into production. Top of mind for you, what are the ways that you're thinking or that you're seeing companies leverage SDG now to actually accelerate their LLM application development rather than just doing it in some sort of way that just because I can, I'm going to do it a way that actually generates business value. Are we seeing this in today's market? Yeah, absolutely. I mean, it's, it's definitely not, you know, we're not at the level where everything needs SDG all the time, but certainly to augment existing data sets, domain-specific data, right? So if you have like a thousand or 2000 examples of say legal data set, well, you can use that to seed a much larger corpus of data. The same thing with things like, especially for like embedding fine tuning, right? Oftentimes we know the corpus that our embedding model has to work well in. And if we know the corpus, then we can use that corpus to help make sure that our embedding model achieves the best performance on that corpus. So it's, uh, you know, this is something that's being used, uh, quite heavily across the board, but especially for those,. And then a lot of the time, whenever we're talking about eval, we're talking about golden data set, golden data set, golden data set, right? Well, until you have a golden data set produced by humans or synthetically aided, and then carried over the final mile by humans, right? Until you have that, you have no golden data set, right? And so we have to start somewhere. And evaluation is the only place we can actually get signal as to how our application is performing. And so using SDG to help us get the first few steps for evaluation is going to really help speed up our prototyping stage and let us move into somewhere close to production. Yeah. And this idea of speeding up, I think is so essential. You know, oftentimes it's like, well, I should wait until I have all the data, right? I should wait until I have it all in a good place. Then I'll do the fine tuning. Then I'll do the pre-training. Then I'll do the domain adaptation. Then I'll do the alignment. My data people are working on it. My data science team is working on it. My mechanical Turk around the world is working on it. The data's not done. I can't get started yet. the world is working on it. The data's not done. I can't get started yet. This to me is the number one thing that I'm hearing from enterprises that that's where I'm always saying, well, what about SDG? I mean, this is to me, the essence is the time savings that you get with SDG, right? I mean, it's kind of like, if you don't have data, how about you SDG it up until you get data at least, right? That's absolutely right. I mean, like, right, so SDG solved the same kind of problems that LLM applications solve, which is this zero data problem, right? And the ability to go quickly from, like you said zero to something right and i i think people often underestimate the power of something compared to nothing even if it's not yet perfect it's going to be able to help you catch uh really simple or straightforward flaws in your application it's going to be able to help make sure that your base use case is really robust, right? It might not be able to get to all of the edge cases and be as, you know, as finalized as you want, but it's going to really help allow something to guide you through those iterations, as opposed to just kind of playing around, right? It lets us have a way that's more rigorous than nothing. And that's extremely valuable, especially as we try to move from the sandbox to the real world. That's right, that's right. Yeah, so if you've got nothing, maybe SDG it up. And the other thing is people always say, well, I can't actually build my application now because I don't have Lama 3.170b set up and I can't expose my data to OpenAI. Well, you can't expose SDG data to OpenAI because it's just SDG data. And so there are benefits to the synthetic data that mimics your data. And there's a lot of ways to encourage you to go from zero to one before you worry about scaling and doing everything perfectly and having all the privacy controls in place. I think this sort of zero to one piece, you can notice, you can use it anywhere. So anywhere where you sort of say, okay, I'm struggling to get started, zero to one, think SDG. Okay. Wiz, we'll have you back in just a little bit for continued discussion. We're going to sort of talk a little bit now about financial services. I just wanted to mention this for those of you in this particular domain, and we're going to try to focus in on specific domain verticals in upcoming events as well. When we talk to our partners, like folks at Langchain and elsewhere, their customers are really in the document heavy business. So this is one of the things that you see across the board, all verticals, but in particular particular you want to think as you're doing things where do i have lots of either text data or you know conversation data that's going to not have to have me crunch a bunch of numbers this is particularly important in the financial services space, but rather allow me to leverage the best of LLMs right off the shelf and quickly get small wins. Now, you know, I want to just sort of mention this contextually that it was about March 2023, the Bloomberg GPT paper came out. So the financial services industry has been out on the sort of edge of trying to do things. This is kind of a this wasn't a synthetic data generation thing. This was a handcrafted, curated, big data set where they were doing this sort of full pre-training from scratch. One of the things you might think about today is we're in sort of the stage of SLMs. If you're building internal LLMs to your financial institution, this domain adapted pre-training, something that's definitely potentially worth looking into where you're sort of not pre-training from scratch, but rather you're sort of augmenting data with more financial services specific stuff like Bloomberg GPT had tried to do not all that long ago. And you can sort of make some progress in that particular space quite fast now in a way that you really couldn't before. In terms of what we're hearing from our partners about financial institutions and what they're really sort of augmenting their people with in terms of tooling is like, there's the research piece, the searching and researching, and then there's the actual analysis piece. And I think the easier piece to sort of do well, it really does depend on what exactly your business is and exactly what you're doing for your customers. But in general, we can be accelerating the research piece and we can be making it easier for analysts to do their job. We can make it easier to get all the data in the right format so that we can quickly accelerate decisions and decision-making in fast-paced environments. And so this is what we're going to kind of focus in on today. This is the one we're hearing about as kind of the place that most of the financial services firms are playing. And it's also hard to kind of do both quantitatively and qualitatively, but we'll show you a way that you can kind of get started with simple documents today. Just one quick use case. This is something that we were listening to over the last year. were listening to over the last year. Scotiabank was telling the story a few weeks ago when we were in Toronto listening to their chief AI officer tell us about what they had gone through. And the Ask AI system that Scotiabank rolled out, they sort of went through this 17 iteration thing with their payments bot. They thought, well, this is the most data we have is on payments. So how about we just build something where we have the most data? And it turned out that was actually really hard. And it was really hard because it was a lot of different types of data, a lot of different structures of data, some very quantitative, not so well suited for off the shelf LLMs. And then in the second sort of use case that they built out, they built out very quickly and produced a lot of business value with was sort of the contact center bot. So they, instead of asking, where do I have the most data? They said, well, what part of our business has employees spending the most time on it? And in that case, they said, oh, contact center, customer support, classic, right? Absolutely classic use case. Right. Absolutely. Classic use case. And they kind of said, well, that. Data is super easy to use. We can create a rag QA chat bot, ask AI kind of thing for our customer support specialists. And it really did start accelerating and saving time for them there. Now, interestingly, on the topic of SDG, neither of these are particularly great to use SDG on, although you could. So I want to have Wiz come back up for just a moment to discuss this. I mean, there are some areas where we want to watch out using SDG, like anytime we have multi-step reasoning or very complex sort of qualitative and quantitative financial stuff, probably not a great use case for SDG. Why? Yeah. I mean, so it's a, it's a great use case for one half of the problem, right? Which is the inputs to these, to these systems. I think this is really, you know, something we want to focus on here is that SDG can always be some part of your pipeline for, you know, say, evaluating or whatever it is that you're going to use it for, but you have to be aware of the limitations so that we don't just blanket throw it at the problem. So in the case of this multi-step reasoning where the output of our system is expected to be quite, you know, quite intense or let's say quite quite involved, then we we might not reach to SDG as a viable tool to produce that complex output. We can still use it to produce the inputs, right? And then we can have our SMEs answer those questions. We can also use, say, you know, our system itself to answer the question and then have our SMEs verify the response or tweak the response to help us get to that final mile of the golden data set. But yeah, definitely SDG by itself without a robust pipeline, which begins to look a lot like your application, right? Is going to be not the first tool you should be reaching for in these very complex use cases. Yeah, yeah. And then, you know, I'm sort of drawn to this other use case that they have, which was on this customer support piece as well. You know, when you have a ton of input output data from customers that your customer support people answered, that's pretty good golden data set right there. Right. I mean, you could augment it with SDG, but like you might not necessarily need to. Right. It might be enough data as it is. If you're feeling like it's low data, maybe you could, but not necessarily required here, right? Yeah, that's right. If you're a legacy business and you've been doing tech support or customer service for the last 15 years, right? You've already got a large history of really valuable, you know, interactions that you can use as data to test your system going forward. Now, you know, say you make changes to your system or you add new content to your knowledge base. You want to test to see how the system does on that. Well, we can extend that data set exactly as you mentioned, Greg, using SDG, using that original data set as our seed data, and then extend it out to these new use cases so that we're covering our bases as they come out. Sure. But when we already have what is, in essence, the most golden data set, it's tough to think that synthetic data is going to be able to get close to that. Mm-hmm. Although if you say, well, it's going to take me seven months to get the golden data set put into the right format, SDG. That's right. That's right. All about speed of development. Absolutely. That's right. Well, Synthwave SDG might come in handy there for you guys in that situation. Okay. All right, Wiz. We'll have you back for the demo shortly. And we are going to now kind of introduce the build, but we're going to do it by telling you a little bit about how we're going to evaluate and then how we're going to evolve the data we need to do these evaluations. the data we need to do these evaluations. So remember, evaluation is just asking, is my app good at this thing I wanted to do? And we're going to build a question answering system for finance documents today. Now, in Langsmith, as in so many other tools out there, many other tools out there. We can simply evaluate using prompts. And at the core of a lot of these evaluations, you'll find exactly that, prompts. So for instance, this is the chain of thought QA evaluator. You are a teacher grading a quiz. You are given a question, the context the question is about, and the student's answer. You are asked to score the student's answer as either correct or incorrect based on the context. Write out in a step-by-step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset. Notice we're given a role and a persona, very specific instructions, very specific outputs, told the LM is being told, hey, think through step by step, make sure that you get it right. And as we continue on here, we're given exactly the input and output format we want. exactly the input and output format we want. And finally, a final kind of grade only based on this factual accuracy, et cetera, et cetera, go ahead and begin. So what we can see is if we zoom in on just a single evaluator here in Langsmith, we see that prompting is really at the core of a lot of these evaluation techniques. Prompting is at the core of so much. And evaluation and synthetic data generation is no different in that way. Now, in Langsmith, there are a couple of different evaluator types. We just saw one of the QA evaluators, the chain of thought QA. There's also more simple evaluators that are so simple, you might not even believe it, but we are going to leverage some of these today. And, you know, there's the QA evaluators, this chain of thought, contextual accuracy is the one we just looked at. We will use that one today in our evaluation, where it essentially is saying, don't just grade as correct, not correct. Don't just make sure you take into account the context for a rag system, but also use chain of thought reasoning before you determine your final answer. Now, if we look at kind of the simple ways we can use criteria evaluators, here's one. It's called accuracy. Hey, how accurate is this prediction? That's literally the prompt. On a scale of one to 10, this would be a criteria evaluator without labels. There's not just accuracy, there's many criteria evaluators within Langsmith, from conciseness to maliciousness to criminality and much more. And you'll notice a lot of these are quite simple. Creativity, is the submission creative, imaginative, or novel, and so on. This is a no-label approach. You could also use a labeled approach that gives the evaluators specific pieces of information to consider. This is going to be essential with our SDG setup today. How accurate is this prediction compared to the reference on a scale of 1 to 10? Here we give the prediction the answer. Here we give the reference the context potentially, right? And we want to compare these two. Actually, in this case, the prediction is sort of the answer generated by our system, and the reference is kind of our golden ground truth answer that we've either collected using a golden data set approach or generated because we don't have a golden data set. And, of course, we can do custom evaluators, and we'll do a little custom one because, you know, we always want to see if what we're building is dope or nope. All of these metrics leverage prompt engineering. As we get into evolving test sets, we want to understand, again, you know, this is just about doing stuff faster, doing stuff more streamlined, getting stuff up and running by using what we already have by bootstrapping. This isn't different than any of the classic computer vision stuff, where we would take the pictures and we would flip it and reverse it, kind of generate a whole bunch of new images, but they were just the same old images. It's a little bit different, right? It's not really different from those image approaches when we're using these synthetic data approaches now. Although we are trying to get better at this. And so to have really great synthetic data, we want to make sure that we're providing a nice kind of wide swath of what we expect to see in production, right? And we can sort of imagine if we're looking for like specific defects in an image, but we don't have enough of the data on those defects. Remember how you would create specific data with just more of those defects? Well, we want to use synthetic data generation in the same way with text. And there's some ways that we can, out of the box, use some kind of cool tools today, and one of which we'll show you. And we can evolve new questions based on old questions. So given a seed question, we can use an evolutionary approach, one that will rewrite the question in a way that enhances the need for reasoning, or one that will modify the question to introduce a conditional element, adding complexity, or one that will rephrase the question in a manner that requires it uses multiple pieces of context from our RAG application to actually answer it correctly. And this rephrasing of the question, whether it's a reasoning or it's a conditioning or it's a multi-context approach, this is at the center of what we're doing when we're doing this synthetic data generation and this evolution of new data so we're going to use a framework from the rag assessment team from ragas that uses this evolution in the center of their puzzle. So there's chunks from our documents, from our rag system. We're gonna make sure those chunks are legit. And we're gonna start with seed questions. The seed questions will then evolve new questions and we'll make sure that they are good to go. Sometimes these things will fail and we'll have to sort of run it back, try to evolve a different question to generate a test data set, essentially from our data that gives us questions and answers that we can use to assess how well we did. So this particular library is built on ideas that came out of the Wizard LM data set that was leveraged to train the wizard lm model and the whole idea of this eval instruct this kind of evolutionary instruction generation was that if i have a bunch of stuff that's not good at instruction following how about i turn it into stuff that is good at training an lm on how to follow instructions. And there was sort of two types of evolving that were used here. In-breadth and in-depth evolving. This is a crazy image. You can't really see anything. Let's zoom in though. If we zoom into the initial instruction here, one plus one equals question mark. You can see how these in-depth evolving techniques, adding of constraints, how to prove that one plus one equals two in the Goldbach conjecture, increase reasoning, so make it more than just this addition. Deepening in what situation is one plus one not equal to two? That's pretty epic. Or we can sort of break out of the frame a little bit and do what's called an in-breadth evolution. But what's happening in the in-depth evolution exactly? Well, we can look behind the scenes and we can see the prompt. Your objective is to rewrite the prompt into a more complex version that makes it a bit harder to handle. Complicate the prompt using the following methods. This is, for instance, the constraints or requirements method. Add more constraints. And you can imagine a similar prompt is happening in deepening and is happening in these different evolutionary steps. So if we go and we ask one plus one to evolve in a way that really makes it something new. We can go from one plus one equals question mark to how many times faster is light than sound in a vacuum to photosynthesis. And this is pretty cool because now we're getting sort of adjacent domains. And so the idea here of this prompt is the new prompt should belong to the same domain, but be even more rare. So it's pretty cool that this works. And what happens is it happens in kind of two steps. The generator does what it does. It generates. And it generates using the seed question. And look at this. It's given context, key phrase, question. The question should be formed using the topic. And so you get these examples, almost bootstrapped few shot examples here, as you're doing this generation. So the generator generates and the critic critiques. The critic is like, hey, based on that new evolved question, assess it for clarity and answerability, and make sure that it's good. This again, another prompt. So what we see is we can see that SDG and the evolution, we can see that eval, and all of this is really coming down to prompting. Is SDG really just all prompt engineering? Well, I'll leave that for you guys to decide. As we get into our final portion of today's event, where we want to evaluate financial documents based on the QA we're able to have with them. Think about how an analyst might look at, for instance, a 10K or 10Q from the recent quarterly report. We might say, well, was it worth dumping a bunch of Apple stock right now? If you're, for instance, Warren Buffett or some other hedge fund or some other money manager trying to make the same decision. In three easy steps, we can create a simple RAG application that allows us to do Q&A against such documents. We can generate test set data using an eval instruct type approach. And we can evaluate using that test data with Langsmith. Let's see how to do it. This high quality 10Q report analysis with SDG. Wiz, show us how it's done, man. Hey, thanks, Greg. Yes, we will go through this notebook here. So basically, the idea is straightforward enough. We're going to generate some synthetic question ground truth pairs. We're going to load them into a Langsmith data set. We're going to evaluate a basic rag chain against the synthetic test data, make changes to our pipeline, and evaluate the modified pipeline. So these last two steps are really what gives you good velocity, right, when you're building LLM applications, since we need to be able to make changes and then evaluate how those changes impacted our application. Without this step, right, without this synthetic test data, we're not able to do these iterations. And that is a shame. Okay, so let's move through the notebook here. First of all, we're going to grab our dependencies. We need LangSmith, LangChain, LangChainCore, LangChain, LangChain, LangChain. We need a bunch of different LangChain submodules. This is to help verify that we have all the tools we're going to need because we're going to be using a few today. First of all, we're going to be using LangSmith both for tracing and for evaluation. Secondly, we're going to be using Together API to power our ragchain and part of our LangSmith evaluation. Then we're going to be using, of course, OpenAI. OpenAI is going to help generate that synthetic data that we can use to test our application. So in order to generate that synthetic data, we need to go ahead and load first our documents using the PyMOO PDF loader. As Greg said, we're going to be using the 10Q filings from Apple. And then we're going to use Ragas's tools to do this. You'll notice that we have this distribution. This distribution is what's going to be able to let us choose between various styles of Qs and As, right? So number one, we have like these simple questions and answers then we have some multi-context that is questions and answers that require multiple contexts and then we have some more complicated reasoning questions and the reason this is good is because we want to be able to determine what the actual you know where the strength in our system lays, where the weaknesses lay. So we're going to be using GBT 4.0 mini as our generator LLM. Our critic LLM is going to be GBT 4.0 and our embeddings is going to use the text embedding 3 small, a default from the OpenAI embeddings thanks to the line chain. This is all we have to do to set this up is simply add our generator with our generator LLM, critical LLM and embeddings, set our distributions, and then we can go ahead and call generator.generate with line chain docs. We'll pass in the documents that we loaded, we'll pass in the number of questions we want to be made, and we'll pass in the distributions they should be in. Notice that we have a bunch of questions, and all these questions, you know, are related to that distribution. So we have some simple questions, like what was the total number of shares purchased? With the answer, the total number of shares purchased. And then we have multi-context questions. And then we have things like this reasoning question. What do forward-looking statements suggest? So these are more complicated reasoning questions. So we create those, and then we can load them into our Langsmith dataset. So first, we'll initialize the Langsmith client, and then, of course, we'll name our dataset, and then we will create that dataset. We're going to iterate through our Regus created data frame here. So this is our test set. We're going to iterate through the rows, and we're going to set the our ragas created data frame here so this is our test set uh we're going to iterate through the rows and we're going to set the inputs to be questions and then the outputs are going to be called answer even though in the ragas uh data frame they're actually called ground truths right langchain is going to expect them to be answered though you can customize it it's a just very ever so so slightly more lines of code. So we won't be doing that today. And then we'll include our context as well because we should. Okay. We're going to send that to that dataset ID. And then we're going to wind up with something that looks a little bit like this, which is going to be a dataset. So what we can look at is the examples in our dataset. And you can see here that we have a number of great examples right what controls should Tim Cook have for accurate financial reporting and we see the answer Tim Cook should have designed just started with that should have designed disclosure controls and procedures to ensure that material information relating to the and on and on right so this is a very uh you know kind of complex question with a fairly complex response which is exactly what we want since that was a reasoning question okay so now that we have our synthetic data set set up just 20 rows but it still is 20 rows we're going to go ahead and build our basic rag chain. Again, we're going to use that same document as our source document here, except this time we're going to chunk it into bite-sized pieces of 500 characters long. We're going to use an embedding model from Together AI. This is going to be the M2 BERT 80 million 8k retrieval, which is a fairly decent embedding model. Then we're going to create our vector store powered by Quadrant, of course. We're going to set that as a retriever. We're going to build our RAG prompt, and then we're going to use meta 3.170b as our LLM of choice today. Again, powered by Together AI. Okay, we're going to set up a rag chain, and we can ask a question, right? So recently, Warren Buffett dumped $90 billion of Apple stock based on the 10K filing report, right? Does Apple seem to be a good financial health? Well, yes. Based on the context, Apple's financial health appears to be good. Provided documents show that the company's net sales have increased or remain relatively flat in various categories. So the RAG system, a little bit perplexed by this decision. All right, well, let's evaluate this, right? So first of all, we're going to use an eval LLM. This is going to be MetaLama 405B Instruct. So this is the big Lama 3.1 model. That's going to be used as partially our evaluator model. We're going to use QA evaluator, context QA evaluator, chain of thought QA evaluator, all from Lang Smith, out of the box. These are evaluators that are going to consider, you know, not just what we've put in in our response but also potentially our output. We're going to use labeled helpfulness. Labeled helpfulness specifically is going to care about what our reference answer was. So this is the ground truth that we generated earlier with RAGIS. So what we're going to ask is the submission helpful to the user taking into account the correct reference answer. So this is going to ask is the submission helpful to the user, taking into account the correct reference answer. So this is going to be important for us to consider. And then, of course, because we're AIM, we've got to do DOPE or NOPE. So we'll go ahead and use the DOPE or NOPE evaluator. And then to kick off evaluation, all we have to do is call evaluate with our RAG chain, pointing out our data set with our desired evaluators, and we're going to give it some metadata to know what it is. We're then going to go ahead and rerun this evaluation, except this time we're going to dope up the prompt. You must answer the question in a dope way. Be cool, right? We're going to use bigger chunks, and we're going to use a better embedding model, and we're going to see how this changes, right? Now, immediately when we ask the same question, does Apple seem to be in good financial health? We see just visually it's more dope. Yes, Apple's financials are looking straight fire. They're rocking a total of, you know, big number, big number. Oh, this is great. Okay. So this is definitely more dope at a glance, but let's see what the evaluation shows. And what we can see is the first of a number here was our original chain and then the second was our uh our new chain that has been expanded you'll notice that across contextual accuracy correctness chain of thought contextual accuracy it's increased as well it is increased in dopeness it's remained steady inness. So we can see now going forward what changes we might want to make in order to improve our application. But for now, we're very pleased to see that most of these metrics did increase across our iteration. And especially we saw massive gains in dopeness. The scales on these numbers are not representative so they're not the same our dopeness went from 0.1 less than 0.1 dopeness all the way to over 70 percent dope and that is uh that's not something that you you know uh you you want to walk away away from without being excited so with that uh i'll pass you guys back to g who will take us to Q&A. But before I do, I've got to ask you to like, comment, and subscribe. Hit that notification bell if you want to see these events. We're live every Wednesday at this time. We're always talking about what's going on in the space and how we can use that with our various tools, discussing latest concepts and code in that AI space. So with that, we'll head back to Greg. All right. So was the Warren Buffett stock dump dope or no? You decide. In conclusion today, we kind of saw that you can leverage SDG anytime you have the low data, the no data, or the just not yet ready data problem. You can go faster with SDG anytime you have the low data, the no data, or the just not yet ready data problem. You can go faster with SDG. And you can leverage it across almost every part of the stack, whether you're talking task-specific behavior, domain adaptation, alignment, evaluation, even more. But you want to ask specifically whether you're using it for the right thing in your domain. We looked at financial services and we can see that we can accelerate knowledge-based development, similar to kind of you imagine we have all this pile of 10 Qs or 10 Ks in a knowledge base. We can accelerate the development, choosing embedding models, making sure that we're answering sufficiently complex questions, using evolution and evaluation techniques together. And we can also stay away from things like that require multi-step reasoning, whether we already have great data for, we really don't want to be selecting SDG for those particular use cases. So we'll go ahead and jump into a few Q and A points here, Wiz. We've got some in the slide already, and we'll get started with these. Rustam asks, can we use the SDG as a form of knowledge distillation from a bigger LLM to a smaller LLM? You know it. Absolutely. Right? Distillation is another excellent use case for SDG. Another part of the stack. There we are. All right. So Anonymous asks, should I create synthetic company documents or synthetic QA for my chatbot using a rag pipeline or both? And how should I use them? Yeah. I mean, the answer is always going to be or both in any question that we're talking about data, data quality, right? So whatever we can do to ensure the highest quality data. So definitely or both here. The basic idea would be, you know, if you're looking to fine tune a betting model, let's say, then you're going to use them in question, context, answer triples. If you're looking to do something like, you know, SFT or task instruction, then creating instructions, or sorry, instruction tuning, and creating instructions that kind of are aware of your company's, you know, knowledge or the domain that your company operates in is going to give you decent results. Yeah. Nice. Nice. Okay. Yeah. I mean, it kind of always depends, right? You know, well, should you take a good hard look at exactly the use case and downscope it, downscope it it, down scope it if you need to do too much SDG to cover too many different things with the same app. Maybe you should be building two different apps. And then Don Branson asks, is it about incrementally raising the bar here? As a domain model evolves, you can use RAG and fine-tuned models to improve the quality of synthetic data. Is it all about incrementally raising the bar? It sure is, Don. That's the secret sauce all the time, right? So if we wait until big changes, then we're waiting, right? So that incremental change is going to be something you want to think about we want to think about making sure that this process is repeatable reproducible so that we can continue to do it as our data evolves as our algorithms evolve as our models evolve right so absolutely this is about a series of small incremental changes and when it comes down to it right if we just sit there waiting for some arbitrary threshold, then that's an amount of time that we're not improving anything at all. And your users, your, you know, whoever's a consumer of your product is going to be best served by seeing some updates a little bit quicker than just waiting for some arbitrary threshold. And finally, Akash asks, how can I generate synthetic data for fine tuning JSON function calling? Trying to compete on that function calling leaderboard. Yes. That's a bit of a specific one. I would say you can use the JSON function calling as an example, and then, or sorry, seed data, and then generate more rows. Asking a code profession LLM, say, to generate functions for various synthetically generated questions, and then using those for various synthetically generated questions, and then using those functions to infer the JSON schema. I think a combination of powerful LLMs here, specifically some tailored for code, is gonna be very useful. And of course, with LLAMA 3.1 having that tool calling API, we're able to get some really, you know, quick iteration on a large volume of these kinds of function calling outputs. So I'd say that's a good place to start. All right. Well, Wiz, that's about all we have in the Q&A. And time went fast today on this big topic. So thanks for joining us and walking us through SDG. All right, everybody. Thank you for joining us on this Wednesday session. We are here every week on YouTube Live. If you enjoyed this content or stuff going on in the chat and you're not yet part of the AI Makerspace community, definitely hop into Discord. We'd love to see you and start following your journey, building, shipping, and sharing with all of us. And then if you're excited about potentially accelerating your LLM application development, like literally right now, we've got our next and biggest, baddest cohort yet coming up starting next Tuesday. The application deadline for cohort four is Friday, August 9th. So if you're interested, go ahead and get those applications in over the next day or two, and you'll have some work to do over the weekend to make sure you get accepted. But we've got a brand new 10-week expanded curriculum. We dropped a link to a quick video you can watch in the chat. And we'd love to hear any questions or comments you have on the way the industry is progressing and our AI engineering bootcamp is keeping up with it. So with that, everybody, thank you for joining us for another YouTube Live Wednesday session. Please do provide any feedback that you have on either the Luma review or in the forms that we share with you in the chat. And until next time, keep building, shipping, and sharing, and we will most certainly do the same. See you on Discord. See you on YouTube. Real soon, everybody. Have a great week. Bye, guys.
Synthetic Data for Enterprise
As LLMs become increasingly vital in technology, selecting the right use cases and pairing them with emerging techniques remains a challenge. However, SDG stands out for its high utility in areas like training, fine-tuning, and evaluation. Join us as we explore how SDG accelerates LLM application development, from enhancing RAG assessments to scaling personas and beyond. We'll dive into the practical applications of SDG, including a detailed case study with LangSmith to test and optimize LLM prototypes in the financial sector. Learn the mechanics of SDG, discuss its implementation in real-world scenarios by top financial services firms, and follow a live code walkthrough to see how you can integrate these strategies into your projects. Whether you're looking to refine your LLM applications or streamline their journey to production, this event will provide the insights and tools necessary to harness the full potential of synthetic data. 


Event page: https://bit.ly/data4enterprise

Have a question for a speaker? Drop them here: 
https://app.sli.do/event/gVRvDR7ESovhXRgcQEXe7e

Speakers: 
Dr. Greg, Co-Founder & CEO AI Makerspace
https://www.linkedin.com/in/gregloughane

The Wiz, Co-Founder & CTO AI Makerspace
https://www.linkedin.com/in/csalexiuk/

Apply for our new AI Engineering Bootcamp on Maven today! 
https://bit.ly/aie1

For team leaders, check out!
https://aimakerspace.io/gen-ai-upskilling-for-teams/

Join our community to start building, shipping, and sharing with us today!
https://discord.gg/RzhvYvAwzA

How'd we do? Share your feedback and suggestions for future events.
https://forms.gle/BramnJL4FLTm8WFW9