 Hi, this is Lance from Lionchain. So OpenAI just released GPT-4.0 or Omni today, which is a pretty exciting release. It reports both significant improvement in non-English languages, much faster and cheaper in the API than the prior state-of-the-art GPT-4, so that's actually really exciting. And it also incorporates multimodality, so both audio-visual as well as text. So that's a really good thing. Now, the question you might ask is, let's say you already have an app, you're using an OpenAI model, like, say, the state-of-the-art GPT-4 Turbo. How do I make a decision about whether or not it's actually safe to upgrade to this model? And then when we talk about safe, that can kind of mean a few different things. So on one hand, you can think about, are there any regressions to the performance of the application itself? Like I have an app already, it's using GPT-4 Turbo, I'm using a bunch of prompts I've already tuned, and do those prompts just translate over to the new model seamlessly? Do they exhibit some odd behavior or regressions that I actually would wanna characterize, right? Anytime I should change the model in your application, you should really wanna investigate, like, okay, how does the performance of the application actually change? So that that's one and also things like user experience now it reports better latency but like what is that actually does actually work in my case so how does the user experience translate if i change my app from for example gpd4 turbo or let's say i was using another app like one of the gpd35.5 variants, given the new cost here, can I make the leap up to now GPT-4 Omni? And what are the implications on the performance of the app itself or things like latency? So those are all things you'd really want to examine to determine whether or not it's actually safe to make this switch for your user. And you can really think about this in three different pieces. I have a data set of examples. For example, in this case, I have a RAG app of input-output pairs that are kind of my ground truth input, my ground truth output, right? So I have a data set. I have an evaluator that I'm going to show you how to define that'll look at my ground truth answers versus my RAG app answers. And I have a RAG app that simply takes in GPT-4 Omni versus Turbo as a user-defined input. That's all I really need to do. And then I'll show you how to use the UI in LankSmith to really dig into the differences, i.e. look at regressions or improvements that come if I upgrade my app to Omni. That's really it. So here's just some code. I'm defining an index here for RAG. I'm taking the language and expression language documentation, which is around 70,000 tokens of context. It's basically a subset of our documentation. And I'm creating a vector store locally from that. Then I'm defining this ragbot class. It's a super simple app. It takes in OpenAI and a model name. Actually doesn't use langchain at all. It's just using the raw OpenAI wrapper, a wrapper we have around the OpenAI API. So it's super simple. Does retrieval, basically does generation with a standard rag prompt. You all can use a local model if you want with the Lama, but in any case, here's all I need to do. I'll define three different functions, which simply instantiate my rag bot with different settings. So here I'll use GPT-4-11-6, GPT-4-turbo, 4-turbo, and then GPT-4-omni, the new model. That's it. so basically i have three different uh functions that all use my little rag bot with different gpd4 versions that's all i need to do um second my data set so i've already created a data set in langsmith you can actually if you go if you go to like if you go to langsmith you can go to your data system testing tab. My data set is defined right here. It's called LSL eval. And I can look at examples. And over in examples, I can actually see here's all my ground truth inputs and outputs. So again, these are inputs and outputs related to the documentation I built my app from. So this is all consistent. So that's really all I need to do in terms of dataset. In terms of evaluator, I'm going to build a custom evaluator that's going to take, see this reference is the reference answer here. So this is for every question I have a reference answer. And it's also for my chain going to get my RAG pipeline prediction. And it's going to compare my reference to the prediction using this prompt right here this is all super transparent um really nice simple way all i have to do is basically use um you'll see down here but this lang smith evaluate function that i can pass this this evaluator function into it's super simple um and that's all i need to do it's basically going to take in my run and my example. So the run is my chain. The example is the example from my data set. It can extract from my run the predicted answer. It extracts from my data set example the reference answer. It compares. I'm using this prompt here. I use LLM as a judge. In this case, I'm using GBD4 turbo as the judge. And it outputs a structured object as grade, which I specify here. And I normalize that. So the raw score is between 10 best, 1 worst. Normalize that by 10. So it's 1 and 0.1. That's all I need to do. Super simple. I run evaluation on my three models right here. Super simple. I run evaluation on my three models right here. And we can then go over and we can look at our data set. It'll now have three experiments. So that's what you see right here. So you can see my experiments are GPT-4-11-6, GPT-4-Turbo, and GPT-4-O. Or sorry, GPT-4-O, OMNI or O. And what you can see here, which is pretty nice, the answer accuracy. So I can see my answer accuracy, this is now the aggregate score, goes from 0.84 up to 0.88. So it does increase, and you actually see that in this plot here, it does increase between my three experiments. So it does appear that OMNI is indeed better than the other two variants in terms of answer accuracy. Now, let's say I also want to look at latency. This is the other major thing that's, in fact, this is really the thing that they really highlight with Omni. I can look at the latency, P50 latency between the three experiments, and I actually can see the latency shows a big difference. So, you know, between GPT-4 Turbo, the prior state of the art in this model, the latency drops actually quite a bit. It looks like maybe a 30% latency drop here. That's really good. You can see my P50 goes from 23 seconds down to 16 seconds. And the answer accuracy, as we showed before, increases. So that's kind of a clear win there. And if the cost benefits carry over, then I would know pretty definitively, this looks like a really safe upgrade for my particular app in terms of latency, in terms of my evaluator accuracy, and in terms of cost, which we actually do have costs typically logged, but we don't have the costs yet in for this model, I believe. And so this just came out today, but you'll have that available to you very soon in Lanxmith as well. So if I want to dig in deeper, I can just click these three experiments. I can go to compare, and this opens up our comparison mode. So here what's pretty cool, I can set a baseline. So in this particular case, I'm going to set 11.6 as my baseline, and I then can compare Turbo and GPT-4.0 or Omni. And what I can see as I go through, I can see all the cases that actually get worse or better. And you can see it summarized at the top. Two get better, four get worse. In this case, six get better, three get worse. So, again, we can see that Omni improves, which we also saw from the aggregate scores. But you can really look granularly at each example and see why it improves. You can click here to actually open up that particular example. You can dig in. Here's the reference input, reference output. Here's my three generations. You can really look granularly and explore whether or not you agree with the evaluator. So this really gives you a nice way to granularly go in, convince yourself that it's safe in your particular case to upgrade to the new GPT-4 Omni. And if I zoom all the way back out, in our particular case, this is just, again, eval set, 20 questions relate to language and expression language. What I can see is the, I'm going to bring up the latency as well, the answer accuracy gets better with GPT-4 Omni, which is great. The latency drops quite a bit, so that's also a win. And if the cost reduction is as reported, then this would be a clear win, a safe upgrade in my particular case. Thanks.
How to evaluate upgrading your app to GPT-4o | LangSmith Evaluations - Part 18
OpenAI recently released GPT-4o, which reports significant improvements in latency and cost. Many users may wonder how to evaluate the effects of upgrading their app to GPT-4o? For example, what latency benefit will users expect to gain and are there any material differences in app performance when I switch to the new GPT-4o model.

Decisions like this are often limited by quality evaluations! Here, we show the process of evaluating GPT-4o on an example RAG app with a 20 question eval set related to LangChain documentation. We show how regression testing in the LangSmith UI allows you to quickly pinpoint examples where GPT-4o shows improvements or regressions over your current app.

GPT-4o docs:
https://openai.com/index/hello-gpt-4o/

LangSmith regression testing UI docs: 
https://docs.smith.langchain.com/old/evaluation/faq/regression-testing

RAG evaluation docs:
https://docs.smith.langchain.com/old/cookbook/testing-examples/rag_eval

Public dataset referenced in the video:
https://smith.langchain.com/public/ea1f6ca5-de52-4d36-bd7b-fde3faa74a70/d

Cookbook referenced in the video:
https://github.com/langchain-ai/langsmith-cookbook/blob/515f4140cb2576ea93051ea5bb4adec652e31893/introduction/langsmith_introduction.ipynb