OpenAI Embeddings (and Controversy?!)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper natural language processing mlnews openai openai embeddings nils reimers beir dataset beir benchmark text similarity neural embeddings gpt-3 embeddings gpt 3 openai api openai gpt embeddings splade sentencebert neural retrieval neural search engine vector search engine inner product search semantic search engine gpt-3 search faiq dataset how good is openai
#mlnews #openai #embeddings

COMMENTS DIRECTLY FROM THE AUTHOR (thanks a lot for reaching out Arvind :) ):
1. The FIQA results you share also have code to reproduce the results in the paper using the API: https://twitter.com/arvind_io/status/1488257004783112192?s=20&t=gB3c79VEX8hGJl6WfZa2iA There's no discrepancy AFAIK.
2. We leave out 6 not 7 BEIR datasets. Results on msmarco, nq and triviaqa are in a separate table (Table 5 in the paper). NQ is part of BEIR too and we didn't want to repeat it. Finally, the 6 datasets we leave out are not readily available and it is common to leave them out in prior work too. For examples, see SPLADE v2 (https://arxiv.org/pdf/2109.10086.pdf) also evaluates on the same 12 BEIR datasets.
3. Finally, I'm now working on time travel so that I can cite papers from the future :)
END COMMENTS FROM THE AUTHOR

OpenAI launches an embeddings endpoint in their API, providing high-dimensional vector embeddings for use in text similarity, text search, and code search. While embeddings are universally recognized as a standard tool to process natural language, people have raised doubts about the quality of OpenAI's embeddings, as one blog post found they are often outperformed by open-source models, which are much smaller and with which embedding would cost a fraction of what OpenAI charges. In this video, we examine the claims made and determine what it all means.

OUTLINE:
0:00 - Intro
0:30 - Sponsor: Weights & Biases
2:20 - What embeddings are available?
3:55 - OpenAI shows promising results
5:25 - How good are the results really?
6:55 - Criticism: Open models might be cheaper and smaller
10:05 - Discrepancies in the results
11:00 - The author's response
11:50 - Putting things into perspective
13:35 - What about real world data?
14:40 - OpenAI's pricing strategy: Why so expensive?

Sponsor: Weights & Biases
https://wandb.me/yannic

Merch: store.ykilcher.com

ERRATA: At 13:20 I say "better", it should be "worse"

References:
https://openai.com/blog/introducing-text-and-code-embeddings/
https://arxiv.org/pdf/2201.10005.pdf
https://beta.openai.com/docs/guides/embeddings/what-are-embeddings
https://beta.openai.com/docs/api-reference/fine-tunes
https://twitter.com/Nils_Reimers/status/1487014195568775173?s=20&amp;amp;amp;amp;amp;amp;t=NBF7D2DYi41346cGM-PQjQ
https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9
https://mobile.twitter.com/arvind_io/status/1487188996774002688
https://twitter.com/gwern/status/1487096484545847299
https://twitter.com/gwern/status/1487156204979855366
https://twitter.com/Nils_Reimers/status/1487216073409716224
https://twitter.com/gwern/status/1470203876209012736
https://www.reddit.com/r/MachineLearning/comments/sew5rl/d_it_seems_openais_new_embedding_models_perform/
https://mobile.twitter.com/arvind_io/status/1488257004783112192
https://mobile.twitter.com/arvind_io/status/1488569644726177796

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, everyone, welcome to a special edition of ML news, we have something to discuss. Open AI just released an embeddings endpoint to their API. This is a company by blog post called introducing text and code embeddings in the Open AI API. Now after the let's call them big successes of GPT three, and codecs, which is the model that powers GitHub scope pilot, Open AI pushes forward into the domain of embeddings. Hold on, this video is sponsored by weights and biases, weights and biases is your one stop shop for all your machine learning needs, it will track your experiments with a single line of code will upload automatically all your logs, all your configurations, everything to your cloud, it will automatically grab all the output, all the metrics, all the configurations of your experiments, and store that in one neat location. So you can see your experiments, you can track them wherever they run, you can compare among the experiments, but you can go further, you can then tune your hyper parameters according to the results of those experiments. And all of this is done automatically in a distributed way, you can literally sit on your toilet on your smartphone and tune your hyper parameters and start new experiments. But it's not only experiment tracking and hyper parameter tuning weights and biases has tools for the entire pipeline of machine learning research from the initial idea up until the deployment and beyond that when you actually want to track what you've deployed weights and biases has cool methods to track all of your data set and their dependencies to each other, as well as your models and all kinds of other artifacts that you might produce a very powerful visualizations for all the inputs and outputs of your pipelines, as well as the models themselves. All of this runs in the cloud. But if you're concerned about privacy, there are options to self host the system is free for personal use and for academics and they have great plans for enterprises, small teams, large teams doesn't matter. So thank you very much weights and biases for sponsoring this video. If you don't know them yet, absolutely check them out. It's free, it'll make your life a whole lot easier. Now let's get into the video. So briefly said an embedding model associates a piece of text with a fixed size vector. The fixed size vector can then be used to do semantic similarity search in high dimensional spaces among other things. They have a toy depiction of these embeddings right here. Now as this clearly shows, furries and football fans are in fact linearly separable. So you know, thanks open AI. In order to get these embeddings, you'd interact with the open API, as you would else you'd instantiate it, you call it you get back a vector, they have three different modes available. One is for text similarity, which essentially means that you can put in pieces of text. And if the vectors are close together, that means the text are in some way similar. The second one is for text search where they have a separate encoder for documents, which are, I guess, longer pieces of content, and queries, which are shorter pieces of content. And the idea is that you would rank document vectors against query vector, and then whichever ones fall closest together, those would be the relevant documents to retrieve for that query. It's a bit similar to text similarity, the differences are in the length of the things that you put into the models, and also a little bit of the semantics, although I don't think there's too much of a difference. The last one is code search, which is essentially the same as text search for code. What's also to be said is that these come in different sizes, Ada being the smallest, and DaVinci being the largest DaVinci is the original 175 billion parameter GPT three model size, they do release a paper along with it on how they train this thing and what the results are. And the brief summary is that in various data sets and various tasks, they do beat previous state of the art results, for example, in linear probe classification, which is where you take embeddings, and then you train just a small linear layer on top with a label data set, they outperform previous state of the art, they also do so in text search tasks in the buyer retrieval benchmark. And lastly, they outperform on code search quite a bit. The paper goes into more details on how the model was trained, they explained that it is a contrastive loss that they've used. Essentially, what you want to do is you want to encode pieces of text through the encoder, and then make similar things closer to each other and negatives, in this case, in batch negatives further apart from each other. This does require quite large batch sizes to actually get an accurate distribution of negatives. But you know, it's open AI, so they can do it. As I said, their models go from 300 million parameters for the smallest to 175 billion for the largest with the embedding dimensions going from 1024 up to a ridiculous 12,288. Now you might think the larger dimension is a good thing. But this is not necessarily the case right here. This is one of the criticisms that's going to come up in a short while. You can also see right here that yeah, indeed, the batch size is pretty large, the paper itself goes into a little bit more detail into the results. And here we kind of see the first scratches in what people are now saying about this model, namely that it doesn't seem to perform that well. Now while these average results that they have presented, mostly from their extra large models do outperform other things is very often that they don't outperform them by that much. And if you actually look in selected tasks, then it's not even clear they're the best model. They seem to compare sometimes to quite outdated baselines. As you can see, these papers are sometimes from 2021. And last I checked, it's 2022. So you know, opening, I get your crap in order. Now by far the biggest controversial point right here is the price. As they say in their documentation, encoding 1000 tokens with a DaVinci model will cost you 60 cents. Now 60 cents doesn't sound like a lot, but corpora often have a lot more than 1000 tokens. Remember that tokens are not even words, they're kind of sub words. And that means that this model is quite expensive. Now this gets drastically cheaper if you go down to the smaller models, as you can see, the query embeddings are already 10 times smaller and Babbage and Ada another factor of eight or so. So pretty shortly, this Twitter thread here blew up by Niels Reimers, who says GPT-3 embeddings by OpenAI was announced this week, I was excited and tested them on 20 datasets. Sadly, they are worse than open models that are 1000 times smaller and running open AI models can be at 1 million times more expensive. This is accompanied by a medium post called open AI GPT-3 text embeddings, really a new state of the art in dense text embeddings, where he leverages a lot of these points that I've said previously, like they seem to not compare to the most recent and most performing baselines and their results don't seem to be that far ahead of the competition, especially if you consider the smaller models and also that they did weird selections of data sets that they've trained on. For example, the buyer benchmark has 18 data sets and they have chosen to just test on 11 of them and report average performance across those 11. So Niels assembled his own benchmark of tasks and tested these models against some openly available models. And the most shocking conclusion is that it seems to be that for some tasks, at least, you can get much better performance with the open models at astonishingly low cost. As you can see in this table here, this lists performance against the cost of encoding 1 million documents, which even for the smallest open AI model costs $800 goes up to $60,000 for the largest one. And on the open models, well, the most expensive tested right here will cost you $6.80 and the best performing one $2.40. Now it is to be said that these prices are probably made such that the largest possible shock effect is achieved. Very often when he mentions prices, he says that, well, this is the cost of like a preemptable t4 GPU, which I guess first of all, you get the difficulty of being preemptable, which you don't get with open AI. And second of all, good luck finding quota for a t4 anywhere on the planet right now. But point taken, the open models can be significantly cheaper. And the blog post explores the results from the paper itself also a bit more, again, pointing out that the advantages aren't that much. And something like point one f1 score, and oftentimes even behind the open models. Another point he makes is that the high dimensionality of the embeddings might actually work against you if you're looking to implement anything, because higher dimensional vectors, if you want to build a search index, for example, they require a much more memory intensive index structure, which will cost you more money. And even disregarding money, searching through a higher dimensional space can be a lot slower than searching through a low dimensional space. And he points out that is not really an option to compress these high dimensional embeddings, they are using something like PCA, as that deteriorates their performance quite quickly. Now the claim is just made right here, but I think he must have some experience or references from somewhere. So I guess that would also count for down sampling methods such as random projections. But I don't know, I guess that's still open out there to try. Now it is to be said that when the author here tried to use the open AI API to reproduce the numbers in the paper, it resulted in different numbers, which makes one wonder, did they change the model since the paper? Or maybe is there something wrong with this evaluation? Now curiously, if I read this correctly, actually, the numbers of the current API used are better than the numbers that are in the paper, which is weird. But also people have pointed out minor issues that can creep in and really destroy your results, such as Gwern right here pointing out that you cannot have new lines in your embedding queries, otherwise the embeddings become almost unusable, which is a thing that open AI discusses in their API documentation. However, Reimer's responded to this and said that yes, indeed, he had replaced the new lines, he'd actually use the exact code that he found in an open AI website snippet. So these results do look pretty legit. In fact, one of the main authors of the paper has put out a response, I guess. I mean, it's not responding to anything. It's just a Twitter thread. But it comes kind of in the light of these criticisms about how they evaluate their embedding models in open AI API. This goes into more detail on the evaluation, mainly reciting points from the paper, but being a little bit more, yeah, we don't always achieve the best results possible than the blog post is because the blog post just shows average numbers and says, well, we're state of the art pretty much everywhere. But if you look into detail a little bit more, the picture becomes a bit more murky. I'll link all the threads here in the description. I think one point to be mentioned right here, which is made by the author here and also by the blog post is that hello, this is Yannick from the future. I've waited on this story a bit because we have some new development, the authors quasi responded again and not really brought anything new to the table, but just put sort of the things being said into context here in that they do point out that on many of the information retrieval, so the search tasks, the embeddings are actually performing really well. And that on zero shot, keep that in mind, including, for example, the FIQA data set where they outperform something like BM25 or other models by a wide margin. On top of that, they also put the cost in perspective saying that for this example data set, and this is a fairly, let's say average data set, the cost of embedding the documents and the queries is $80. So the blog post always compared costs of embedding X many millions of tokens. But if you go to actual data set, yes, the embeddings are still going to be more expensive, but the absolute cost might actually not be as much as the blog post might seem. Of course, that depends entirely on how large your data set is. But spending 80 bucks for a 62% relative improvement seems to be a nice deal. So it seems to really depend on the data set at hand, and you might have to try it out on a subset of your data. This was then greeted by a response response, saying that, yes, but the much smaller model and much cheaper model is just point one of a score better than the largest GPT-3 model. So Niels asked why the evaluation was just done on 11 out of the 18 data sets, we don't have a response yet to that, but it's been a week, so I don't expect we'll get one. And that is where it stands currently back to Yannick in the past. In their experience, these embeddings seem to do quite well when you have to transfer them to a new domain. A lot of these openly available models, they are trained on specific data sets, you know, with specific benchmarks in mind and all of that. So they kind of come from the academic world for the academic world, and therefore might overperform even on a different data set, it is still a clean data set that has been assembled kind of to be a benchmark and so on. While what OpenAI is saying that if we take these embeddings and actually go to the real world, our customers see big improvements in their own applications. Now, of course, there's no way to verify that. And the blog posts lists three examples of customers saying, Oh, look, they are able to find like six to 10 times more relevant examples for something or they pump their performance from 64% to 89%. Again, there's no way to verify that but I wouldn't actually be surprised if that is the case. Real world data is a lot more messy than any of the academic data sets. And therefore, I guess only trying it out will actually tell you whether it's useful or not. I do have to wonder about the price though. Like there are two possibilities essentially. One OpenAI has done market research and so on. And this is what they think people will pay for this. Like this is how much value they think they bring with their API. Or on the other hand, this is kind of their operating cost plus some margin to make the shareholders happy. Now I really can't tell apparently they do have customers. So someone must be willing to pay all of this. On the other hand, it does seem outrageously expensive for such a small improvement, at least in these academic data sets. So let me know what you think is this even profitable for OpenAI? Like does anyone have any estimates on what it costs them to develop these new models and to keep them running? It must be massive endeavor. In any case, that was it for the special episode of ML news. Merch is still available. And I'll see you next time. Bye bye.
 Hello, everyone, welcome to a special edition of ML news, we have something to discuss.
 Open AI just released an embeddings endpoint to their API.
 This is a company by blog post called introducing text and code embeddings in the Open AI API.
 Now after the let's call them big successes of GPT three, and codecs, which is the model
 that powers GitHub scope pilot, Open AI pushes forward into the domain of embeddings.
 Hold on, this video is sponsored by weights and biases, weights and biases is your one
 stop shop for all your machine learning needs, it will track your experiments with a single
 line of code will upload automatically all your logs, all your configurations, everything
 to your cloud, it will automatically grab all the output, all the metrics, all the configurations
 of your experiments, and store that in one neat location.
 So you can see your experiments, you can track them wherever they run, you can compare among
 the experiments, but you can go further, you can then tune your hyper parameters according
 to the results of those experiments.
 And all of this is done automatically in a distributed way, you can literally sit on
 your toilet on your smartphone and tune your hyper parameters and start new experiments.
 But it's not only experiment tracking and hyper parameter tuning weights and biases
 has tools for the entire pipeline of machine learning research from the initial idea up
 until the deployment and beyond that when you actually want to track what you've deployed
 weights and biases has cool methods to track all of your data set and their dependencies
 to each other, as well as your models and all kinds of other artifacts that you might
 produce a very powerful visualizations for all the inputs and outputs of your pipelines,
 as well as the models themselves.
 All of this runs in the cloud.
 But if you're concerned about privacy, there are options to self host the system is free
 for personal use and for academics and they have great plans for enterprises, small teams,
 large teams doesn't matter.
 So thank you very much weights and biases for sponsoring this video.
 If you don't know them yet, absolutely check them out.
 It's free, it'll make your life a whole lot easier.
 Now let's get into the video.
 So briefly said an embedding model associates a piece of text with a fixed size vector.
 The fixed size vector can then be used to do semantic similarity search in high dimensional
 spaces among other things.
 They have a toy depiction of these embeddings right here.
 Now as this clearly shows, furries and football fans are in fact linearly separable.
 So you know, thanks open AI.
 In order to get these embeddings, you'd interact with the open API, as you would else you'd
 instantiate it, you call it you get back a vector, they have three different modes available.
 One is for text similarity, which essentially means that you can put in pieces of text.
 And if the vectors are close together, that means the text are in some way similar.
 The second one is for text search where they have a separate encoder for documents, which
 are, I guess, longer pieces of content, and queries, which are shorter pieces of content.
 And the idea is that you would rank document vectors against query vector, and then whichever
 ones fall closest together, those would be the relevant documents to retrieve for that
 query.
 It's a bit similar to text similarity, the differences are in the length of the things
 that you put into the models, and also a little bit of the semantics, although I don't think
 there's too much of a difference.
 The last one is code search, which is essentially the same as text search for code.
 What's also to be said is that these come in different sizes, Ada being the smallest,
 and DaVinci being the largest DaVinci is the original 175 billion parameter GPT three model
 size, they do release a paper along with it on how they train this thing and what the
 results are.
 And the brief summary is that in various data sets and various tasks, they do beat previous
 state of the art results, for example, in linear probe classification, which is where
 you take embeddings, and then you train just a small linear layer on top with a label data
 set, they outperform previous state of the art, they also do so in text search tasks
 in the buyer retrieval benchmark.
 And lastly, they outperform on code search quite a bit.
 The paper goes into more details on how the model was trained, they explained that it
 is a contrastive loss that they've used.
 Essentially, what you want to do is you want to encode pieces of text through the encoder,
 and then make similar things closer to each other and negatives, in this case, in batch
 negatives further apart from each other.
 This does require quite large batch sizes to actually get an accurate distribution of
 negatives.
 But you know, it's open AI, so they can do it.
 As I said, their models go from 300 million parameters for the smallest to 175 billion
 for the largest with the embedding dimensions going from 1024 up to a ridiculous 12,288.
 Now you might think the larger dimension is a good thing.
 But this is not necessarily the case right here.
 This is one of the criticisms that's going to come up in a short while.
 You can also see right here that yeah, indeed, the batch size is pretty large, the paper
 itself goes into a little bit more detail into the results.
 And here we kind of see the first scratches in what people are now saying about this model,
 namely that it doesn't seem to perform that well.
 Now while these average results that they have presented, mostly from their extra large
 models do outperform other things is very often that they don't outperform them by
 that much.
 And if you actually look in selected tasks, then it's not even clear they're the best
 model.
 They seem to compare sometimes to quite outdated baselines.
 As you can see, these papers are sometimes from 2021.
 And last I checked, it's 2022.
 So you know, opening, I get your crap in order.
 Now by far the biggest controversial point right here is the price.
 As they say in their documentation, encoding 1000 tokens with a DaVinci model will cost
 you 60 cents.
 Now 60 cents doesn't sound like a lot, but corpora often have a lot more than 1000 tokens.
 Remember that tokens are not even words, they're kind of sub words.
 And that means that this model is quite expensive.
 Now this gets drastically cheaper if you go down to the smaller models, as you can see,
 the query embeddings are already 10 times smaller and Babbage and Ada another factor
 of eight or so.
 So pretty shortly, this Twitter thread here blew up by Niels Reimers, who says GPT-3 embeddings
 by OpenAI was announced this week, I was excited and tested them on 20 datasets.
 Sadly, they are worse than open models that are 1000 times smaller and running open AI
 models can be at 1 million times more expensive.
 This is accompanied by a medium post called open AI GPT-3 text embeddings, really a new
 state of the art in dense text embeddings, where he leverages a lot of these points that
 I've said previously, like they seem to not compare to the most recent and most performing
 baselines and their results don't seem to be that far ahead of the competition, especially
 if you consider the smaller models and also that they did weird selections of data sets
 that they've trained on.
 For example, the buyer benchmark has 18 data sets and they have chosen to just test on
 11 of them and report average performance across those 11.
 So Niels assembled his own benchmark of tasks and tested these models against some openly
 available models.
 And the most shocking conclusion is that it seems to be that for some tasks, at least,
 you can get much better performance with the open models at astonishingly low cost.
 As you can see in this table here, this lists performance against the cost of encoding 1
 million documents, which even for the smallest open AI model costs $800 goes up to $60,000
 for the largest one.
 And on the open models, well, the most expensive tested right here will cost you $6.80 and
 the best performing one $2.40.
 Now it is to be said that these prices are probably made such that the largest possible
 shock effect is achieved.
 Very often when he mentions prices, he says that, well, this is the cost of like a preemptable
 t4 GPU, which I guess first of all, you get the difficulty of being preemptable, which
 you don't get with open AI.
 And second of all, good luck finding quota for a t4 anywhere on the planet right now.
 But point taken, the open models can be significantly cheaper.
 And the blog post explores the results from the paper itself also a bit more, again, pointing
 out that the advantages aren't that much.
 And something like point one f1 score, and oftentimes even behind the open models.
 Another point he makes is that the high dimensionality of the embeddings might actually work against
 you if you're looking to implement anything, because higher dimensional vectors, if you
 want to build a search index, for example, they require a much more memory intensive
 index structure, which will cost you more money.
 And even disregarding money, searching through a higher dimensional space can be a lot slower
 than searching through a low dimensional space.
 And he points out that is not really an option to compress these high dimensional embeddings,
 they are using something like PCA, as that deteriorates their performance quite quickly.
 Now the claim is just made right here, but I think he must have some experience or references
 from somewhere.
 So I guess that would also count for down sampling methods such as random projections.
 But I don't know, I guess that's still open out there to try.
 Now it is to be said that when the author here tried to use the open AI API to reproduce
 the numbers in the paper, it resulted in different numbers, which makes one wonder, did they
 change the model since the paper?
 Or maybe is there something wrong with this evaluation?
 Now curiously, if I read this correctly, actually, the numbers of the current API used are better
 than the numbers that are in the paper, which is weird.
 But also people have pointed out minor issues that can creep in and really destroy your
 results, such as Gwern right here pointing out that you cannot have new lines in your
 embedding queries, otherwise the embeddings become almost unusable, which is a thing that
 open AI discusses in their API documentation.
 However, Reimer's responded to this and said that yes, indeed, he had replaced the new
 lines, he'd actually use the exact code that he found in an open AI website snippet.
 So these results do look pretty legit.
 In fact, one of the main authors of the paper has put out a response, I guess.
 I mean, it's not responding to anything.
 It's just a Twitter thread.
 But it comes kind of in the light of these criticisms about how they evaluate their embedding
 models in open AI API.
 This goes into more detail on the evaluation, mainly reciting points from the paper, but
 being a little bit more, yeah, we don't always achieve the best results possible than the
 blog post is because the blog post just shows average numbers and says, well, we're state
 of the art pretty much everywhere.
 But if you look into detail a little bit more, the picture becomes a bit more murky.
 I'll link all the threads here in the description.
 I think one point to be mentioned right here, which is made by the author here and also
 by the blog post is that hello, this is Yannick from the future.
 I've waited on this story a bit because we have some new development, the authors quasi
 responded again and not really brought anything new to the table, but just put sort of the
 things being said into context here in that they do point out that on many of the information
 retrieval, so the search tasks, the embeddings are actually performing really well.
 And that on zero shot, keep that in mind, including, for example, the FIQA data set
 where they outperform something like BM25 or other models by a wide margin.
 On top of that, they also put the cost in perspective saying that for this example data
 set, and this is a fairly, let's say average data set, the cost of embedding the documents
 and the queries is $80.
 So the blog post always compared costs of embedding X many millions of tokens.
 But if you go to actual data set, yes, the embeddings are still going to be more expensive,
 but the absolute cost might actually not be as much as the blog post might seem.
 Of course, that depends entirely on how large your data set is.
 But spending 80 bucks for a 62% relative improvement seems to be a nice deal.
 So it seems to really depend on the data set at hand, and you might have to try it out
 on a subset of your data.
 This was then greeted by a response response, saying that, yes, but the much smaller model
 and much cheaper model is just point one of a score better than the largest GPT-3 model.
 So Niels asked why the evaluation was just done on 11 out of the 18 data sets, we don't
 have a response yet to that, but it's been a week, so I don't expect we'll get one.
 And that is where it stands currently back to Yannick in the past.
 In their experience, these embeddings seem to do quite well when you have to transfer
 them to a new domain.
 A lot of these openly available models, they are trained on specific data sets, you know,
 with specific benchmarks in mind and all of that.
 So they kind of come from the academic world for the academic world, and therefore might
 overperform even on a different data set, it is still a clean data set that has been
 assembled kind of to be a benchmark and so on.
 While what OpenAI is saying that if we take these embeddings and actually go to the real
 world, our customers see big improvements in their own applications.
 Now, of course, there's no way to verify that.
 And the blog posts lists three examples of customers saying, Oh, look, they are able
 to find like six to 10 times more relevant examples for something or they pump their
 performance from 64% to 89%.
 Again, there's no way to verify that but I wouldn't actually be surprised if that is
 the case.
 Real world data is a lot more messy than any of the academic data sets.
 And therefore, I guess only trying it out will actually tell you whether it's useful
 or not.
 I do have to wonder about the price though.
 Like there are two possibilities essentially.
 One OpenAI has done market research and so on.
 And this is what they think people will pay for this.
 Like this is how much value they think they bring with their API.
 Or on the other hand, this is kind of their operating cost plus some margin to make the
 shareholders happy.
 Now I really can't tell apparently they do have customers.
 So someone must be willing to pay all of this.
 On the other hand, it does seem outrageously expensive for such a small improvement, at
 least in these academic data sets.
 So let me know what you think is this even profitable for OpenAI?
 Like does anyone have any estimates on what it costs them to develop these new models
 and to keep them running?
 It must be massive endeavor.
 In any case, that was it for the special episode of ML news.
 Merch is still available.
 And I'll see you next time.
 Bye bye.