[ML News] Cedille French Language Model | YOU Search Engine | AI Finds Profitable MEME TOKENS
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper mlnews machine learning news ai news ml news cedille french language model gpt-j gpt j eleuther ai you you search you search engine richard socher meme tokens dogecoin finu finu token wmt facebook wmt multilingual wmt multilingual machine translation machin translation deepmind arnheim arnheim yann lecun alibaba damo acessibe eyebobs lawsuit
#mlnews #cedille #wmt

Only the greatest of news from the world of Machine Learning.

OUTLINE:
0:00 - Sponsor: Weights & Biases
1:50 - Cedille - French Language Model
3:55 - Facebook AI Multilingual model wins WMT
5:50 - YOU private search engine
10:35 - DeepMind's Open-Source Arnheim
12:10 - Company sued for using AI to make website more accessible
18:05 - Alibaba DAMO Academy creates 10 Trillion M6 model
21:15 - AMD MI200 Family
22:30 - State of AI report 2021
24:15 - Andrew Ng's Landing AI raises 57M
25:40 - Cerebras raises 250M
26:45 - Microsoft's Varuna: Scalable Training of Huge Models
28:15 - Laura Ruis reproduces Extrapolation Paper
29:05 - Ian Charnas' Real-Life Punchout
30:00 - Helpful Things
33:10 - AI finds profitable Meme-Tokens
34:55 - This Sneaker Does Not Exist

Sponsor: Weights & Biases
https://wandb.com

References:
Cedille - French Language Model
https://en.cedille.ai/
https://github.com/coteries/cedille-ai
https://app.cedille.ai/
https://en.wikipedia.org/wiki/Cedilla

Facebook AI Multilingual model wins WMT
https://ai.facebook.com/blog/the-first-ever-multilingual-model-to-win-wmt-beating-out-bilingual-models/

YOU private search engine
https://you.com/
https://youdotcom.notion.site/FAQ-8c871d6c99d84e02955fda772a1da8d4

DeepMind's Open-Source Arnheim
https://deepmind.com/research/open-source/open-source-arnheim-a-learnable-visual-grammar-for-generating-paintings
https://twitter.com/OriolVinyalsML/status/1459231774068854785
https://github.com/deepmind/arnheim
https://colab.research.google.com/github/deepmind/arnheim/blob/master/arnheim_2.ipynb

Company sued for using AI to make website more accessible
https://www.wired.com/story/company-tapped-ai-website-landed-court/
https://archive.ph/kdvOM

Alibaba DAMO Academy creates 10 Trillion M6 model
https://pandaily.com/alibaba-damo-academy-creates-worlds-largest-ai-pre-training-model-with-parameters-far-exceeding-google-and-microsoft/
https://www.infoq.cn/article/xIX9lekuuLcXewc5iphF

AMD MI200 Family
https://www.anandtech.com/show/17054/amd-announces-instinct-mi200-accelerator-family-cdna2-exacale-servers?utm_source=pocket_mylist

State of AI report 2021
https://www.stateof.ai/?utm_source=pocket_mylist

Andrew Ng's Landing AI raises 57M
https://techcrunch.com/2021/11/08/landing-ai-machine-learning-operations-tools/
https://www.forbes.com/sites/bernardmarr/2021/11/09/landing-ai-unlocking-the-power-of-data-centric-artificial-intelligence/
https://landing.ai/platform/

Cerebras raises 250M
https://cerebras.net/news/cerebras-systems-raises-250m-in-funding-for-over-4b-valuation-to-advance-the-future-of-artificial-intelligence-compute/
https://cerebras.net/news/cerebras-systems-announces-worlds-first-brain-scale-artificial-intelligence-solution/

Microsoft's Varuna: Scalable Training of Huge Models
https://syncedreview.com/2021/11/10/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-142/

Laura Ruis reproduces Extrapolation Paper
https://lauraruis.github.io/2021/11/06/extra.html?utm_source=pocket_mylist
https://github.com/LauraRuis

Ian Charnas' Real-Life Punchout
https://www.reddit.com/r/MachineLearning/comments/qpenkt/project_google_movenet_realtime_pose_estimation/
https://www.youtube.com/watch?v=07JibJJVNp8

Helpful Things
https://www.marktechpost.com/2021/11/05/google-ai-introduces-goemotions-an-nlp-dataset-for-fine-grained-emotion-classification/
https://pair-code.github.io/lit/demos/
https://github.com/pair-code/lit
https://www.reddit.com/r/MachineLearning/comments/qsrdyk/p_texttoimage_rudalle_kandinsky_xxl_12_billion/
https://twitter.com/yeemachine/status/1457779633449934848?utm_source=pocket_mylist
https://github.com/yeemachine/kalidokit

AI finds profitable Meme-Tokens
https://finance.yahoo.com/news/artificial-intelligence-now-makes-possible-104800931.html
https://finu.co/

This Sneaker Does Not Exist
https://thissneakerdoesnotexist.com/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hold on, this video is sponsored by weights and biases. Weights and biases is your one stop shop for all your machine learning needs. It will track your experiments with a single line of code will upload automatically all your logs, all your configurations, everything to your cloud, it will automatically grab all the output, all the metrics, all the configurations of your experiments and store that in one neat location. So you can see your experiments, you can track them wherever they run, you can compare among the experiments, but you can go further, you can then tune your hyper parameters according to the results of those experiments. And all of this is done automatically in a distributed way, you can literally sit on your toilet on your smartphone and tune your hyper parameters and start new experiments. But it's not only experiment tracking and hyper parameter tuning weights and biases has tools for the entire pipeline of machine learning research from the initial idea up until the deployment and beyond that when you actually want to track what you've deployed weights and biases has cool methods to track all of your data set and their dependencies to each other, as well as your models and all kinds of other artifacts that you might produce a very powerful visualizations for all the inputs and outputs of your pipelines, as well as the models themselves. All of this runs in the cloud. But if you're concerned about privacy, there are options to self host the system is free for personal use and for academics and they have great plans for enterprises, small teams, large teams doesn't matter. So thank you very much weights and biases for sponsoring this video. If you don't know them yet, absolutely check them out. It's free, it'll make your life a whole lot easier. Now let's get into the video. Welcome, welcome to ml news. Let's dive into our first story group of researchers based in Switzerland have trained city which is a French language model. This is a model based on GPTJ to 6 billion parameter model that is a language model in French. The headline is write French without speaking French, which is pretty much a recipe of how I passed high school. So the cool thing about this is that it can do the tasks that you're used to from things like GPT three, but with a special focus on French. So it achieves a better perplexity on French text than GPT three, apparently lower toxicity, whatever that means is better at translating from and to French, and it's better at various other NLP tasks out of the box. And if you don't know what city means, city is this little thing that French people put at the bottom of some of their letters, also some other languages as I am being told, but just quite annoying because you never know where on the keyboard it is. So being quite annoying seems like a great name for a French language model. The cool thing is not only is the model open source, you can download a checkpoint, the code is open source, but also you can play with it directly in the browser, there's a little app and there are a bunch of prompts that are already built in, for example, classification of some stuff like what is FedEx FedEx is logistics company that is correct, Amazon is an e commerce and technology company that is all correct. Now my French is limited to be honest, Jay, who Bly a more baguette, Joe sweet, the Zola, I think it means I lost my baguette and I'm very sad. The model says meme see, ne pas d'explication logi. I don't have a logical explanation for why I lost my baguette. Is it maybe I forgot my baguette? I don't know. Well, in any case, it's a French language model, you get it. What is interesting is that among the parameters, it says that a German one is coming soon. So keep an eye out for that. Facebook AI on their blog says the first ever multi lingual model to win a WMT beating out bilingual models. So WMT is this yearly competition essentially to do machine translation. This is a corpus of data sets, but then also every year the competition hosts human expert translators that rate the translations of the machine translation systems. So the machines aren't able to hyper optimize on the data sets, but really have to please the humans. Now first thing, why is this in the AR VR category? I don't know. In any case, it's quite remarkable because one would think that given that all the tasks are bilingual, that bilingual models that can be tailored to one specific language pair would be ahead right here. But as Facebook AI shows, because multi lingual models can ingest essentially much more data into them. So the French English translations are also informed by the German data that comes in. And because it's able to make use of so much more data, it can in the end outperform models that have been trained for particular language pairs. Now multi lingual ity is not the only thing that's good about this model. The machine translation community has over the years accrued various tricks such as back translation to make use of monolingual data, ensembling, and so on. So this is really an engineering effort. But it's cool to see this overlap point where for the first time ever a single multi lingual model is better than many, many bilingual models. And that's excellent, not only because it's higher performing, but it also means that it provides us easier access to work with languages that have very low resources that maybe are only spoken by a very small amount of people or that have no written form at all, like Swiss German, for example. So excellent development, there is a paper, the code is available. And if you want to learn all the tricks, give it a read. You is a new search engine that has been launched by Richard soccer previously the head of AI at Salesforce. And this is supposed to be a direct competitor to the Google search engine, you advertise itself as the private search engine that summarizes the web for you. So there's two promises here, privacy, and summarization in whatever form, they say it helps you get things done, get news, check GitHub, compose a tweet all from your search engine, for whatever reason you want to compose a tweet from your search engine. But there you go. There's a big emphasis on privacy, you can choose between a personalized or a truly private experience, you.com never sells your data to advertisers. And also they promise no ad targeting. Now actually, when you sign up, the first thing that they want to make you do is install an extension. If I click this button, it leads me straight into the Chrome Web Store. So I'm gonna take this with a grain of salt right here, someone promises me privacy, no targeting, and so on. No, unless this is provably the case, I'm not going to trust any of those promises. So the second big selling point is this summarize the web. And I was intrigued by that, like how is this search engine gonna summarize the web for me, this sounds really cool. So I tried out a bunch of things like, okay, they said I could check news, for example. All right, news, let me zoom out a little bit here. So the interface that you gives you is this kind of grouped interface. So there are web results on top right here, there is a section for news. And then there are various of these subcategories right here. But honestly, I don't see any summarization like any summarize the web for me. So let me search for something I would like to have summarized Abraham Lincoln and the Civil War. No, it just gives me the Wikipedia page and a bunch of web results and a bunch of Reddit results and a bunch of these quick facts right here. Now one thing seems to be these shortcuts these apps right here. So there are various apps, for example, the quick facts app, which we have down here, or I guess the Wikipedia app, which is up here. So the search engine seems to be such that other developers can come in and write apps for it. So you can install apps in your search engine. And those will take up one of these bars. As you can see, there are apps for archive, Walmart, all kinds of things. There's also one for GitHub. But I haven't seen yet this summarize what was Lincoln's role in the Civil War. Again, I just get a bunch of search results. I don't see exactly how summarize the web should be anything like this. So I was also exploring a bit of different features right here. For example, compose a tweet. So I tried this previously, it actually told me to sign into Twitter. So apparently, you can write tweets from here how to sort a list in Python. Now this gets into a little bit more interesting things, they have plugins for Stack Overflow, and also W three schools. So they show the results from these sites in quite nice cards with snippets and so on. For Stack Overflow, there's also a sidebar, which for some reason doesn't show up right now. There's also this code completion engine right here. So I entered how to sort a list of strings in Python. And it gives me a bunch of code completion that are apparently generated by some sort of code model. I mean, that's fine. So I've tried a bunch of things with this search engine, but I really haven't seen this summarize the web for you in any particular way. This seems to be a search engine where other people can write apps for it. And then it'll probably send your search query to those apps. And the apps can give you useful results. Now honestly, it seems like a big benefit for sort of like the big websites right here. For example, W three schools is integrated prominently as you can see, tutorials point is integrated prominently Coursera Stack Overflow, this is specifically for code. But if you look at the other apps that exists, it's essentially all the big websites. So I'm not sure if I actually want this in a search engine, I generally want the most relevant things and I don't necessarily want the relevant things from the biggest sites while I see the potential of integrating all of these things into my search engine is not that useful. Honestly, how many heads does a Hydra have? I quite like this shortcut right here. So this little G, it brings you to this website that you might have heard of. But this is also a pretty good search engine. And it generally gives me the stuff I'm looking for. That being said, you is public now and it is in beta. So you know, give it a little slack until it really full out. And maybe this concept of having many apps integrate into your searches provided by other people and not all by the same company will be something for the future. Who knows? DeepMind releases open source Arnheim, a learnable visual grammar for generating paintings. So bouncing off of the success of people experimenting with clip models such as VQ GAN plus clip or clip guided diffusion, or any of these models that generate stunning images by using clip, DeepMind has done something a little bit different, namely, instead of using a GAN or a diffusion, they are using a what they call a visual grammar. So you're able to give some primitives to the model of how it can compose an image. And then we'll use that in order to please clip in order to do clip guided image generation. So one application of this is, for example, here, you give the model a grammar of brush strokes. So you tell it that it can do some brush strokes in some various ways, various colors, various thicknesses, and so on. You give a bunch of optimization parameters, and it can generate pictures from textual descriptions. It looks pretty cool, I have to say, and it has some nice controllable parameters. Here, you can see the evolution of such a picture as it develops over time, you can see that the model refines on how exactly it lays its brush strokes until it reaches a final conclusion. photo realistic chicken. Yeah. So the code is available along with two colabs where you can try it out for yourself. Oriole vinyls has tweeted out this picture right here of young LeCun made up entirely of MNIST digits. So the model here hasn't gotten brushstrokes as option to perform drawings, but just MNIST digits in various colors. And you know, it looks pretty sweet. So check out paper and code and blog post and give it a try. Wired writes this company tapped AI for its website and landed in court. So this is an article about a company that is being sued because its website does not conform to the accessibility standards of the W three C consortium. The company in question is called IBOBS. And it used this other company called accessibility to make its site more accessible. Now, if you make a website, you can do that with various frameworks. But in order to make it accessible to for example, visually impaired people, you need to annotate the various parts of your website with their meaning you give alt text to images, you define an order of focus, for example, in forms, they should all be navigatable by your keyboard by using the tab key, for example, auto complete should work and so on and so on. Now there are already many tools to help you with that. But it's still a very, very high workload for developers to ship out websites that are also accessible to all the people that want to use them. So this company accessibility says that it can simplify the work of making websites accessible to people with impaired vision or other challenges are replacing a costly manual process with an automated state of the art AI technology. However, this technology doesn't seem to be working all that well in all cases, which is something you could expect, right? So this whole article doesn't only detail this case, but it says it's a growing trend in recent years, companies use these AI softwares to make their websites more accessible, these don't work really well, that makes the websites worse for visually impaired people compared to when manual labor is used to do the same thing and so on. Noteworthy the guidelines that you have to comply with is more than 100 pages when printed, it includes such things as alt text for images and video, clear use of contrast and color, ensuring that features like forms and menus are navigatable using only keyboard without the use of a mouse or finger and so on. Now safe to say this is a difficult problem, right? Of course, AI solutions are going to be largely subpar when it comes to this compared to really dedicated humans doing this. However, they're probably going to be better than just the developers doing it on the side as they're coding the website under time pressure. And they're certainly going to be better than nothing at all. Like I get it, the web sucks for visually impaired people interacting with a medium that is this visual when your visuals don't work is bad, it's it's a bad experience. And it largens the divide between people who have good vision and people who have poor vision, I get this and they also get that we want to make an effort as a society to include visually impaired people more to make websites more accessible, and so on. But I don't see when the standard has become that unless a solution works 100% of the time, a lawsuit should be filed. Like surely having a crappy AI annotated website for visually impaired people is better than not having an annotated website at all. On the other hand, that you can absolutely see that if we as a society decide, well, just use the AI tool for this, then companies are going to opt for that and actually avoid putting in the work of making websites really accessible. So it is a hard problem. And I don't have the clear answer for this. But I would certainly say that AI technology can help it's better than nothing. It gives you sort of a lower bound on accessibility on a website, even if there are some mistakes, because humans make mistakes too. But here is what I find funny. There is apparently a document a sort of petition where researchers and companies and so on can put their name to ask other people to ask other companies not to use these AI tools. It says signers include contributor to W3C guidelines and employees at Microsoft, Apple and Google. Automated detection and repair of accessibility problems is not reliable enough to bring a site into compliance, the document says, accusing some vendors of deceptive marketing. And here it comes. The site was started by Karl Groves, founder of the accessibility consultancy, Tenon.io, who provided a withering 35 page analysis of accessories software to Murphy's lawsuit against iBobs. So iBobs, me being sued, they used accessibility software. And now this Tenon.ai Karl Groves has written a 35 page analysis of this software. Groves said he surveyed a total of about 1000 pages from 50 websites using the startup that's accessibility technology and found a median of 2300 violations of W3C guidelines for each site. Here it comes. Groves says that this is a significant undercount because most of the guidelines can only be checked by an expert manual analysis. So wait, did I understand this correctly? Did you analyze 1000 websites and you either automatically or by non expert humans figured out a lower bound on the number of violations to the standards. And that's not actually the standards, but it's a lower bound and therefore it's better than nothing at all. Really, you did that. And you provide that as evidence into a lawsuit. Hypocrite, hypocrite, hypocrite, hypocrite, hypocrite, hypocrite. In his report to AccessiBee, Groves cited an image of a model wearing a white dress for sale on an e commerce site. The alternative text provided apparently generated by AccessiBee's technology was grass nature and summer. Oh no, an anecdote. Wow. And there you have it. The true story here is that complaining is easier than doing and we'll always be able to write articles about AI systems that don't work 100% yet. As I said, I don't have the definite solution to this problem. It is a hard problem. It's a balance between pushing technology and making it accessible to all the people there are. But how funny that's all I'm gonna say. PanDaily reports Alibaba Damo Academy creates world's largest AI pre-training model with parameters far exceeding Google and Microsoft. Right, so this is about a model called M6 by Alibaba Damo Academy. And the parameter count in these models is one trillion to 10 trillion, far exceeding the trillion level model previously released by Google and Microsoft becoming the world's largest AI pre-training model. I found another article by info queue right here, which I had to translate from Chinese. So M6 stands for multimodality to multimodality multitask mega transformer, M6. That's why it's called M6. And the whole article is like an homage to Chinese research. The real thing that's hailed here as a breakthrough is the efficiency by which people can train these models. But the parameter count is a little bit tricky, because this model uses a mixture of experts architecture, which we can assume maybe to be sparse. And therefore a sparse model with a trillion parameters is not necessarily better than a dense model with 900 billion parameters, given that the network is only activated sparsely. At this point, we don't exactly know what we know is that the model is multimodal, which means it processes images, it processes text and so on. One of the invention highlighted by the article is what they call grouped mixture of experts or what they call expert prototyping. They say it's so that different groups of mixtures of experts can increase the expression space of the model without changing the parameter scale. No idea what that means. So they tout that it can create more high resolution pictures like Dalí can create fashion, as you see here can create textual descriptions, find similar images and so on. Alibaba achieved efficient training of the trillion m six model with only 480 v 100 cards, reducing energy consumption by more than 80%. And the efficiency is increased by nearly 11 times. Right. So this seems to be the real achievement right here, the investigation into efficient model training. As I said, we don't exactly have better data right now, at least I wasn't able to find it. What is a bit deceptive is that the title says that the model has 10 times the number of neurons as humans. So apparently it has what trillion parameters and the human brain has 86 billion neurons yet. Of course, the number of neurons is not equal to the number of parameters for that you need the synapses in the brain, which are more than 125 trillion. So no, your parameter count is not larger than human parameter count quite yet. And even if we get there, it's probably not going to perform as well as humans just because you have that many parameters. If you people figure out any more about this model, link it down below in the comments. Let me know the scale and design of this models are amazing. This looks like a manifesto to the gradual growth of many Chinese AI research organizations. Yeah, they kick your butt if you don't write this info queue. This is like there's a guy in the corner being like, this is great, isn't it? Isn't it? Excellent journalism, everyone. On on tech rights, AMD announces the instinct mi 200 accelerator family. So this is AMD is newest incursion into the GPU space, they say they can connect whatever they learn from building CPUs and GPUs together. And I honestly don't understand many of the things that are said right here, or what's supposed to be special. So as far as I can understand it, one thing that's special is that their machines have like one memory for the CPUs and the GPUs, which eliminates the need of shipping data back and forth, which is one of the main bottlenecks in applications when using GPUs. Maybe I'm wrong. Another thing is that the individual parts that you can put together into bigger parts into bigger servers, they are connected using super duper fast whatever connections instead of PCI connections, which makes things yet even faster. So for their biggest servers, they have 95.7 teraflops of floating point 32 matrix operations. And if you go to FP 16, they have 383 teraflops. I'm being told that's a really good thing. I have no idea. But if you're interested in this, if you maybe want to buy one, get in touch with AMD, please sponsor me. The State of the AI report 2021 is out. This is produced by AI investors Nathan Benight and Ian Hogarth. So actually, it's October 12. So this thing has been out for a while, but forgive me for only reporting on this right now. So as it says, these two people are investors. So they naturally have a distinct look onto the field, which is interesting, right. So it's divided into various sections like research trends. It does quite a good job of summarizing sort of what's going on currently in research, where talent is in which countries at which universities and so on. Notably, China seems to be rising quite a bit in pumping out AI graduates, as you can see right here. Now, it's a quite a lengthy presentation. But what's really interesting is their predictions for the next 12 months. For example, transformers replace recurrent networks to learn world models with which RL agents surpass human performance in large and rich game environments. That's quite a specific prediction, but could actually be true, right? Small transformers and CNN hybrid models match current state of the art on ImageNet top one accuracy with 10 times fewer parameters. A new AGI focused research company is formed with significant backing and a roadmap that's focused on a sector vertical eg developer tools for life science. Well, I guess them being investors, they can just make that happen and then claim their prediction was correct. But it's pretty cool. I'm excited to follow which ones will actually work out and where they are completely wrong. Probably they're under betting most of these things quite a bit. But you know, that's just my opinion. If you're interested in the more general report, as I said, it's quite interesting carries together a lot of data into a neat little package. TechCrunch writes landing AI brings in 57 million US dollars for its machine learning operations tools. So landing AI is a company started by Andrew Ng and has just raised $57 million to build essentially an ML ops platform. They're doing what they're calling data centric AI. And the whole idea is that things like convolution neural networks or in general machine learning models, they're as easy to build as downloading a bit of code from GitHub and running it on your data set. So the real challenge nowadays is really to get the data set to a quality where you can actually train some good model on it. So their product is essentially this data manager and data labeler tool where it helps professionals really label the data. This is all geared towards manufacturing. So here you'd label cracks or dents or whatnot in newly manufactured phones and then you train your model on very little data. And that's then supposed to give you a nice detector for classifying further manufacturing defects. So their idea isn't necessarily to build one big model that's going to solve all the problems but to provide the different industry players in manufacturing with the tools to build their own models from very little but very high quality data so they can essentially get their expertise into these models. I guess that's not a dumb idea. If you're a manufacturer, maybe you want to try landing lens. Another startup that has raised a lot of money is cerebras raising 250 million US dollars or an over 4 billion US dollar valuation. So cerebras builds these really big chips that are geared specifically towards AI computation. Now, as I said before, I have no clue what's going on in these chip manufacturing processes and what's important and whatnot. But these are apparently really, really big chips and everything's connected to everything in memory super fast and memory is with the compute and yada yada yada, what you need to know is that there are indeed other players than Nvidia or AMD in the space of providing compute solutions for AI. And that's a good thing. And maybe at some point, cerebras will come away from their giant chips and actually also make consumer products. Who knows? If that happens, it's going to be good for all of us. And if they stay in the big chip server world, I think it's still good for us because all of the cloud compute might get cheaper because there's just more competition. Speaking of cheap synced rights, Microsoft India proposes Varuna, a scalable and low cost training of massive deep learning model system. So this is essentially an engineering paper that details how you can train big models on cheap and unreliable hardware. So the system uses both data parallelism as well as model pipelining. So you split up your data batches across different machines, you also split up your models across different machines. And if you do that in a smart way, you can achieve actual big throughput. So usually big models have to be trained on what they call hyper clusters, which means clusters that have very fast interconnect because in order to do something like an all reduce if you have to do layer normalization or batch normalization, I don't remember which one it is, sometimes you need to send data around, sometimes you need to send gradients around, and that costs a lot of compute and bandwidth and so on. So it's very interesting to see that these researchers are able to compete with these big hyper cluster training procedures and essentially bring that down to a heterogeneous clusters of spot instances that can die at any time. It's cool to see that AI training of these big models becomes something like a Kubernetes cluster where you can just add machines and the system will reconfigure itself to make optimal use of the machines however fast they may be connected and however long they might be up. So if you're looking for a cheap way to train a 200 billion parameter model, then this might be the way to go. Okay, here is a shout out to a few places. So the first shout out is to Laura Ruiz, his website where she replicates a bunch of things in young Lacan's and others papers called learning in high dimension always amounts to extrapolation. It's a very technical paper and Laura does a great job here, not only replicating the experiments in here, but providing really nice background and reasons and also the code that she uses to do everything. So I just thought this was really neat interleaving plots, code, math, and so on and really going through all of this. And in the end, actually being able to reproduce the plots of the papers, Yippee, there it is so beautiful, very reproduced much similar. If you want to follow Laura, definitely check out our website or GitHub. This is absolutely beautiful photo Laura. Good job. Right, another cool project is real life punch out by Ian Charnas. This is a really well made video about using body tracking models and pairing them up with punch out the N64 game. So you can actually play this in the browser, it tracks your arms, and you can punch using various boxing moves and play punch out. Not only that, but Ian actually went ahead and bought many cartridges of the game, as you can see in the background right here. And if you play it in the browser, it will actually use one of those cartridges because using just a ROM downloaded from the internet would violate the licensing agreements. So every game you play is essentially corresponding to a real life cartridge. As I said, the video is done extremely well. It's a fun video to watch. Or if you simply want to try it out, you can go to Ian's website and just play it by yourself. Nothing to install runs in the browser. Excellent. Alright, so this is the section where I provide some helpful things. First helpful thing market tech post writes Google AI introduces go emotions and NLP data set for fine grained emotion classification. I've actually shown this in last week's weights and biases ad if you have followed the weights and biases ads. But this is a data set where Reddit comments are annotated with one of I believe 28 different emotions contained in the comments. It's not only one emotion per comment, but technically any emotion could or could not appear in any comment. In total, there are 58,000 Reddit comments classified into on its 27 emotion categories 12 positive 11 negative four ambiguous and one neutral with that adds up to 28. I was right. So the data set creation process detailed here is detailing how they went about it, how they went about balancing the data, paying attention to the fact that Reddit isn't exactly a good replica of the entire world and so on. If you're interested, you can give this article a read, you can also look at the paper that goes along with the data set and you can use the data set if you want to try out your hand at emotion detection. I have to say it's gotten a bit tired to see NLP tutorials always doing sort of semantic classification where it's just positive or negative and this might just provide a little bit of a more challenging task here has this language interpretability tool it's open source and it's for visualizing and understanding NLP models. This provides various things you can look at embedding spaces of NLP tasks, it can analyze things like classification, regression, looking at attention heads, analyzing parts of the input, which parts are important for which things and so on. All in all, it's quite a rich tool and I encourage you to check it out if you're into language interpretability. Or if you want to just check out how your models do the things they're doing code is available tool is available. Okay, last week, we've reported on a rudali the Russian Dalí model. And now apparently the large model is available for download as one Reddit comment says, or much rather the edit of the comment says that the availability is on December 1. So expect that soon. machine on Twitter says after a year in dev, I'm happy to release the core of my Vtuber apps. Now Vtubers are special sort of things that I have never really touched on. But this seems to be a large community that transforms their body movements onto digital anime avatars, as you can see right here. So this also uses body pose tracking and apparently also face tracking in order to make your avatar do as you're doing code is available. And it's not only sort of for face and upper body, but you can also track your entire body movements and map them onto characters as you can see right here, it can do facial point tracking such that it really replicates your facial expressions. So there's never been a better time to become a Vtuber. Check out Khalid o kit on GitHub if you're interested. There's an article by Newsfile Corporation on Yahoo Finance that writes that artificial intelligence now makes it possible for investors to find promising new hidden gem meme tokens automatically. This isn't necessarily what you think you think while there's a company that tells me which meme tokens are good so I can buy it. No, no, no, no, no, no, no. See, this is an actual token itself. So you put money into the token, and then the token selects projects in which the money is to be invested. These projects it says are automatically selected using a special AI based sniper bot. So the AI will look at all the meme tokens, the dodge and the Shiba enu and the squid game tokens, and it will predict which ones will go up and then it will take all the money that is invested into the Finu token, put it into those tokens and then pay out the winnings to the holders of the Finu token. I mean, look at this for an enhanced version of this graphic, please. Yes, I want an enhanced version. Oh, wow, that's enhanced. That that is that is so hands. Absolutely. Currently, there is a website for this and it says vote for Finu help the price pump and hit the back there is a dodge. Okay people who want to make a quick buck using meme tokens that have absolutely no value whatsoever, are encouraged to buy a meme token. Excellent. Now I'm not saying this can't be done. Mean tokens are essentially like fashion that there's no reason why this particular that particular fashion should be in or out next year and yet it still happens and there might be ways to predict it. But still, whether or not this is the way to go can't tell. So I've mentioned this shoe does not exist last week. But there's also this sneaker does not exist. Look at that. And this is pretty cool. So this is a grid of AI generated sneakers, you can click on one, right, and then you can apparently edit that sneaker. So you can go normal to futuristic, you can go high creativity, that's very creative. You can change up the colors a little bit. Very cool, very functional. Look at that one. Yeah, futuristic, creative, light color. I mean, it's not super futuristic. But yeah, so shout out to this sneaker does not exist.com. Check it out. And that was already it for this week's ML news. I hope you had fun hit subscribe if you liked it. We're only 105,900,000 subscribers behind PewDiePie. We can totally catch them. If we really do our jobs, tell three people they're going to tell three people is going to be fine. See you next Monday. Bye bye.
 Hold on, this video is sponsored by weights and biases. Weights and biases is your one
 stop shop for all your machine learning needs. It will track your experiments with a single
 line of code will upload automatically all your logs, all your configurations, everything
 to your cloud, it will automatically grab all the output, all the metrics, all the configurations
 of your experiments and store that in one neat location. So you can see your experiments,
 you can track them wherever they run, you can compare among the experiments, but you
 can go further, you can then tune your hyper parameters according to the results of those
 experiments. And all of this is done automatically in a distributed way, you can literally sit
 on your toilet on your smartphone and tune your hyper parameters and start new experiments.
 But it's not only experiment tracking and hyper parameter tuning weights and biases
 has tools for the entire pipeline of machine learning research from the initial idea up
 until the deployment and beyond that when you actually want to track what you've deployed
 weights and biases has cool methods to track all of your data set and their dependencies
 to each other, as well as your models and all kinds of other artifacts that you might
 produce a very powerful visualizations for all the inputs and outputs of your pipelines,
 as well as the models themselves. All of this runs in the cloud. But if you're concerned
 about privacy, there are options to self host the system is free for personal use and for
 academics and they have great plans for enterprises, small teams, large teams doesn't matter. So
 thank you very much weights and biases for sponsoring this video. If you don't know them
 yet, absolutely check them out. It's free, it'll make your life a whole lot easier. Now
 let's get into the video. Welcome, welcome to ml news. Let's dive into our first story
 group of researchers based in Switzerland have trained city which is a French language
 model. This is a model based on GPTJ to 6 billion parameter model that is a language model in
 French. The headline is write French without speaking French, which is pretty much a recipe
 of how I passed high school. So the cool thing about this is that it can do the tasks that
 you're used to from things like GPT three, but with a special focus on French. So it
 achieves a better perplexity on French text than GPT three, apparently lower toxicity,
 whatever that means is better at translating from and to French, and it's better at various
 other NLP tasks out of the box. And if you don't know what city means, city is this little
 thing that French people put at the bottom of some of their letters, also some other
 languages as I am being told, but just quite annoying because you never know where on the
 keyboard it is. So being quite annoying seems like a great name for a French language model.
 The cool thing is not only is the model open source, you can download a checkpoint, the
 code is open source, but also you can play with it directly in the browser, there's a
 little app and there are a bunch of prompts that are already built in, for example, classification
 of some stuff like what is FedEx FedEx is logistics company that is correct, Amazon
 is an e commerce and technology company that is all correct. Now my French is limited to
 be honest, Jay, who Bly a more baguette, Joe sweet, the Zola, I think it means I lost my
 baguette and I'm very sad. The model says meme see, ne pas d'explication logi. I don't
 have a logical explanation for why I lost my baguette. Is it maybe I forgot my baguette?
 I don't know. Well, in any case, it's a French language model, you get it. What is interesting
 is that among the parameters, it says that a German one is coming soon. So keep an eye
 out for that. Facebook AI on their blog says the first ever multi lingual model to win
 a WMT beating out bilingual models. So WMT is this yearly competition essentially to
 do machine translation. This is a corpus of data sets, but then also every year the competition
 hosts human expert translators that rate the translations of the machine translation systems.
 So the machines aren't able to hyper optimize on the data sets, but really have to please
 the humans. Now first thing, why is this in the AR VR category? I don't know. In any case,
 it's quite remarkable because one would think that given that all the tasks are bilingual,
 that bilingual models that can be tailored to one specific language pair would be ahead
 right here. But as Facebook AI shows, because multi lingual models can ingest essentially
 much more data into them. So the French English translations are also informed by the German
 data that comes in. And because it's able to make use of so much more data, it can in
 the end outperform models that have been trained for particular language pairs. Now multi lingual
 ity is not the only thing that's good about this model. The machine translation community
 has over the years accrued various tricks such as back translation to make use of monolingual
 data, ensembling, and so on. So this is really an engineering effort. But it's cool to see
 this overlap point where for the first time ever a single multi lingual model is better
 than many, many bilingual models. And that's excellent, not only because it's higher performing,
 but it also means that it provides us easier access to work with languages that have very
 low resources that maybe are only spoken by a very small amount of people or that have
 no written form at all, like Swiss German, for example. So excellent development, there
 is a paper, the code is available. And if you want to learn all the tricks, give it
 a read.
 You is a new search engine that has been launched by Richard soccer previously the head of AI
 at Salesforce. And this is supposed to be a direct competitor to the Google search engine,
 you advertise itself as the private search engine that summarizes the web for you. So
 there's two promises here, privacy, and summarization in whatever form, they say it helps you get
 things done, get news, check GitHub, compose a tweet all from your search engine, for whatever
 reason you want to compose a tweet from your search engine. But there you go. There's a
 big emphasis on privacy, you can choose between a personalized or a truly private experience,
 you.com never sells your data to advertisers. And also they promise no ad targeting. Now
 actually, when you sign up, the first thing that they want to make you do is install an
 extension. If I click this button, it leads me straight into the Chrome Web Store. So
 I'm gonna take this with a grain of salt right here, someone promises me privacy, no targeting,
 and so on. No, unless this is provably the case, I'm not going to trust any of those
 promises. So the second big selling point is this summarize the web. And I was intrigued
 by that, like how is this search engine gonna summarize the web for me, this sounds really
 cool. So I tried out a bunch of things like, okay, they said I could check news, for example.
 All right, news, let me zoom out a little bit here. So the interface that you gives
 you is this kind of grouped interface. So there are web results on top right here, there
 is a section for news. And then there are various of these subcategories right here.
 But honestly, I don't see any summarization like any summarize the web for me. So let
 me search for something I would like to have summarized Abraham Lincoln and the Civil War.
 No, it just gives me the Wikipedia page and a bunch of web results and a bunch of Reddit
 results and a bunch of these quick facts right here. Now one thing seems to be these shortcuts
 these apps right here. So there are various apps, for example, the quick facts app, which
 we have down here, or I guess the Wikipedia app, which is up here. So the search engine
 seems to be such that other developers can come in and write apps for it. So you can
 install apps in your search engine. And those will take up one of these bars. As you can
 see, there are apps for archive, Walmart, all kinds of things. There's also one for
 GitHub. But I haven't seen yet this summarize what was Lincoln's role in the Civil War.
 Again, I just get a bunch of search results. I don't see exactly how summarize the web
 should be anything like this. So I was also exploring a bit of different features right
 here. For example, compose a tweet. So I tried this previously, it actually told me to sign
 into Twitter. So apparently, you can write tweets from here how to sort a list in Python.
 Now this gets into a little bit more interesting things, they have plugins for Stack Overflow,
 and also W three schools. So they show the results from these sites in quite nice cards
 with snippets and so on. For Stack Overflow, there's also a sidebar, which for some reason
 doesn't show up right now. There's also this code completion engine right here. So I entered
 how to sort a list of strings in Python. And it gives me a bunch of code completion that
 are apparently generated by some sort of code model. I mean, that's fine. So I've tried
 a bunch of things with this search engine, but I really haven't seen this summarize the
 web for you in any particular way. This seems to be a search engine where other people can
 write apps for it. And then it'll probably send your search query to those apps. And
 the apps can give you useful results. Now honestly, it seems like a big benefit for
 sort of like the big websites right here. For example, W three schools is integrated
 prominently as you can see, tutorials point is integrated prominently Coursera Stack Overflow,
 this is specifically for code. But if you look at the other apps that exists, it's essentially
 all the big websites. So I'm not sure if I actually want this in a search engine, I generally
 want the most relevant things and I don't necessarily want the relevant things from
 the biggest sites while I see the potential of integrating all of these things into my
 search engine is not that useful. Honestly, how many heads does a Hydra have? I quite
 like this shortcut right here. So this little G, it brings you to this website that you
 might have heard of. But this is also a pretty good search engine. And it generally gives
 me the stuff I'm looking for. That being said, you is public now and it is in beta. So you
 know, give it a little slack until it really full out. And maybe this concept of having
 many apps integrate into your searches provided by other people and not all by the same company
 will be something for the future. Who knows? DeepMind releases open source Arnheim, a learnable
 visual grammar for generating paintings. So bouncing off of the success of people experimenting
 with clip models such as VQ GAN plus clip or clip guided diffusion, or any of these models
 that generate stunning images by using clip, DeepMind has done something a little bit different,
 namely, instead of using a GAN or a diffusion, they are using a what they call a visual grammar.
 So you're able to give some primitives to the model of how it can compose an image.
 And then we'll use that in order to please clip in order to do clip guided image generation.
 So one application of this is, for example, here, you give the model a grammar of brush
 strokes. So you tell it that it can do some brush strokes in some various ways, various
 colors, various thicknesses, and so on. You give a bunch of optimization parameters, and
 it can generate pictures from textual descriptions. It looks pretty cool, I have to say, and it
 has some nice controllable parameters. Here, you can see the evolution of such a picture
 as it develops over time, you can see that the model refines on how exactly it lays its
 brush strokes until it reaches a final conclusion. photo realistic chicken. Yeah. So the code
 is available along with two colabs where you can try it out for yourself. Oriole vinyls
 has tweeted out this picture right here of young LeCun made up entirely of MNIST digits.
 So the model here hasn't gotten brushstrokes as option to perform drawings, but just MNIST
 digits in various colors. And you know, it looks pretty sweet. So check out paper and
 code and blog post and give it a try. Wired writes this company tapped AI for its website
 and landed in court. So this is an article about a company that is being sued because
 its website does not conform to the accessibility standards of the W three C consortium. The
 company in question is called IBOBS. And it used this other company called accessibility
 to make its site more accessible. Now, if you make a website, you can do that with various
 frameworks. But in order to make it accessible to for example, visually impaired people,
 you need to annotate the various parts of your website with their meaning you give alt
 text to images, you define an order of focus, for example, in forms, they should all be
 navigatable by your keyboard by using the tab key, for example, auto complete should
 work and so on and so on. Now there are already many tools to help you with that. But it's
 still a very, very high workload for developers to ship out websites that are also accessible
 to all the people that want to use them. So this company accessibility says that it can
 simplify the work of making websites accessible to people with impaired vision or other challenges
 are replacing a costly manual process with an automated state of the art AI technology.
 However, this technology doesn't seem to be working all that well in all cases, which
 is something you could expect, right? So this whole article doesn't only detail this case,
 but it says it's a growing trend in recent years, companies use these AI softwares to
 make their websites more accessible, these don't work really well, that makes the websites
 worse for visually impaired people compared to when manual labor is used to do the same
 thing and so on. Noteworthy the guidelines that you have to comply with is more than
 100 pages when printed, it includes such things as alt text for images and video, clear use
 of contrast and color, ensuring that features like forms and menus are navigatable using
 only keyboard without the use of a mouse or finger and so on. Now safe to say this is
 a difficult problem, right? Of course, AI solutions are going to be largely subpar when
 it comes to this compared to really dedicated humans doing this. However, they're probably
 going to be better than just the developers doing it on the side as they're coding the
 website under time pressure. And they're certainly going to be better than nothing at all. Like
 I get it, the web sucks for visually impaired people interacting with a medium that is this
 visual when your visuals don't work is bad, it's it's a bad experience. And it largens
 the divide between people who have good vision and people who have poor vision, I get this
 and they also get that we want to make an effort as a society to include visually impaired
 people more to make websites more accessible, and so on. But I don't see when the standard
 has become that unless a solution works 100% of the time, a lawsuit should be filed. Like
 surely having a crappy AI annotated website for visually impaired people is better than
 not having an annotated website at all. On the other hand, that you can absolutely see
 that if we as a society decide, well, just use the AI tool for this, then companies are
 going to opt for that and actually avoid putting in the work of making websites really accessible.
 So it is a hard problem. And I don't have the clear answer for this. But I would certainly
 say that AI technology can help it's better than nothing. It gives you sort of a lower
 bound on accessibility on a website, even if there are some mistakes, because humans
 make mistakes too. But here is what I find funny. There is apparently a document a sort
 of petition where researchers and companies and so on can put their name to ask other
 people to ask other companies not to use these AI tools. It says signers include contributor
 to W3C guidelines and employees at Microsoft, Apple and Google. Automated detection and
 repair of accessibility problems is not reliable enough to bring a site into compliance, the
 document says, accusing some vendors of deceptive marketing. And here it comes. The site was
 started by Karl Groves, founder of the accessibility consultancy, Tenon.io, who provided a withering
 35 page analysis of accessories software to Murphy's lawsuit against iBobs. So iBobs,
 me being sued, they used accessibility software. And now this Tenon.ai Karl Groves has written
 a 35 page analysis of this software. Groves said he surveyed a total of about 1000 pages
 from 50 websites using the startup that's accessibility technology and found a median
 of 2300 violations of W3C guidelines for each site. Here it comes. Groves says that this
 is a significant undercount because most of the guidelines can only be checked by an expert
 manual analysis. So wait, did I understand this correctly? Did you analyze 1000 websites
 and you either automatically or by non expert humans figured out a lower bound on the number
 of violations to the standards. And that's not actually the standards, but it's a lower
 bound and therefore it's better than nothing at all. Really, you did that. And you provide
 that as evidence into a lawsuit. Hypocrite, hypocrite, hypocrite, hypocrite, hypocrite,
 hypocrite. In his report to AccessiBee, Groves cited an image of a model wearing a white
 dress for sale on an e commerce site. The alternative text provided apparently generated by AccessiBee's
 technology was grass nature and summer. Oh no, an anecdote. Wow. And there you have it.
 The true story here is that complaining is easier than doing and we'll always be able
 to write articles about AI systems that don't work 100% yet. As I said, I don't have the
 definite solution to this problem. It is a hard problem. It's a balance between pushing
 technology and making it accessible to all the people there are. But how funny that's
 all I'm gonna say. PanDaily reports Alibaba Damo Academy creates world's largest AI pre-training
 model with parameters far exceeding Google and Microsoft. Right, so this is about a model
 called M6 by Alibaba Damo Academy. And the parameter count in these models is one trillion
 to 10 trillion, far exceeding the trillion level model previously released by Google
 and Microsoft becoming the world's largest AI pre-training model. I found another article
 by info queue right here, which I had to translate from Chinese. So M6 stands for multimodality
 to multimodality multitask mega transformer, M6. That's why it's called M6. And the whole
 article is like an homage to Chinese research. The real thing that's hailed here as a breakthrough
 is the efficiency by which people can train these models. But the parameter count is a
 little bit tricky, because this model uses a mixture of experts architecture, which we
 can assume maybe to be sparse. And therefore a sparse model with a trillion parameters
 is not necessarily better than a dense model with 900 billion parameters, given that the
 network is only activated sparsely. At this point, we don't exactly know what we know
 is that the model is multimodal, which means it processes images, it processes text and
 so on. One of the invention highlighted by the article is what they call grouped mixture
 of experts or what they call expert prototyping. They say it's so that different groups of
 mixtures of experts can increase the expression space of the model without changing the parameter
 scale. No idea what that means. So they tout that it can create more high resolution pictures
 like Dalí can create fashion, as you see here can create textual descriptions, find
 similar images and so on. Alibaba achieved efficient training of the trillion m six model
 with only 480 v 100 cards, reducing energy consumption by more than 80%. And the efficiency
 is increased by nearly 11 times. Right. So this seems to be the real achievement right
 here, the investigation into efficient model training. As I said, we don't exactly have
 better data right now, at least I wasn't able to find it. What is a bit deceptive is that
 the title says that the model has 10 times the number of neurons as humans. So apparently
 it has what trillion parameters and the human brain has 86 billion neurons yet. Of course,
 the number of neurons is not equal to the number of parameters for that you need the
 synapses in the brain, which are more than 125 trillion. So no, your parameter count
 is not larger than human parameter count quite yet. And even if we get there, it's probably
 not going to perform as well as humans just because you have that many parameters. If
 you people figure out any more about this model, link it down below in the comments.
 Let me know the scale and design of this models are amazing. This looks like a manifesto to
 the gradual growth of many Chinese AI research organizations. Yeah, they kick your butt if
 you don't write this info queue. This is like there's a guy in the corner being like, this
 is great, isn't it? Isn't it? Excellent journalism, everyone.
 On on tech rights, AMD announces the instinct mi 200 accelerator family. So this is AMD
 is newest incursion into the GPU space, they say they can connect whatever they learn from
 building CPUs and GPUs together. And I honestly don't understand many of the things that are
 said right here, or what's supposed to be special. So as far as I can understand it, one thing
 that's special is that their machines have like one memory for the CPUs and the GPUs,
 which eliminates the need of shipping data back and forth, which is one of the main bottlenecks
 in applications when using GPUs. Maybe I'm wrong. Another thing is that the individual
 parts that you can put together into bigger parts into bigger servers, they are connected
 using super duper fast whatever connections instead of PCI connections, which makes things
 yet even faster. So for their biggest servers, they have 95.7 teraflops of floating point
 32 matrix operations. And if you go to FP 16, they have 383 teraflops. I'm being told
 that's a really good thing. I have no idea. But if you're interested in this, if you maybe
 want to buy one, get in touch with AMD, please sponsor me.
 The State of the AI report 2021 is out. This is produced by AI investors Nathan Benight and
 Ian Hogarth. So actually, it's October 12. So this thing has been out for a while, but
 forgive me for only reporting on this right now. So as it says, these two people are investors.
 So they naturally have a distinct look onto the field, which is interesting, right. So
 it's divided into various sections like research trends. It does quite a good job of summarizing
 sort of what's going on currently in research, where talent is in which countries at which
 universities and so on. Notably, China seems to be rising quite a bit in pumping out AI
 graduates, as you can see right here. Now, it's a quite a lengthy presentation. But what's
 really interesting is their predictions for the next 12 months. For example, transformers
 replace recurrent networks to learn world models with which RL agents surpass human performance
 in large and rich game environments. That's quite a specific prediction, but could actually
 be true, right? Small transformers and CNN hybrid models match current state of the art
 on ImageNet top one accuracy with 10 times fewer parameters. A new AGI focused research
 company is formed with significant backing and a roadmap that's focused on a sector vertical
 eg developer tools for life science. Well, I guess them being investors, they can just
 make that happen and then claim their prediction was correct. But it's pretty cool. I'm excited
 to follow which ones will actually work out and where they are completely wrong. Probably
 they're under betting most of these things quite a bit. But you know, that's just my
 opinion. If you're interested in the more general report, as I said, it's quite interesting
 carries together a lot of data into a neat little package. TechCrunch writes landing
 AI brings in 57 million US dollars for its machine learning operations tools. So landing
 AI is a company started by Andrew Ng and has just raised $57 million to build essentially
 an ML ops platform. They're doing what they're calling data centric AI. And the whole idea
 is that things like convolution neural networks or in general machine learning models, they're
 as easy to build as downloading a bit of code from GitHub and running it on your data set.
 So the real challenge nowadays is really to get the data set to a quality where you can
 actually train some good model on it. So their product is essentially this data manager and
 data labeler tool where it helps professionals really label the data. This is all geared
 towards manufacturing. So here you'd label cracks or dents or whatnot in newly manufactured
 phones and then you train your model on very little data. And that's then supposed to give
 you a nice detector for classifying further manufacturing defects. So their idea isn't
 necessarily to build one big model that's going to solve all the problems but to provide
 the different industry players in manufacturing with the tools to build their own models from
 very little but very high quality data so they can essentially get their expertise into
 these models. I guess that's not a dumb idea. If you're a manufacturer, maybe you want to
 try landing lens. Another startup that has raised a lot of money is cerebras raising
 250 million US dollars or an over 4 billion US dollar valuation. So cerebras builds these
 really big chips that are geared specifically towards AI computation. Now, as I said before,
 I have no clue what's going on in these chip manufacturing processes and what's important
 and whatnot. But these are apparently really, really big chips and everything's connected
 to everything in memory super fast and memory is with the compute and yada yada yada, what
 you need to know is that there are indeed other players than Nvidia or AMD in the space
 of providing compute solutions for AI. And that's a good thing. And maybe at some point,
 cerebras will come away from their giant chips and actually also make consumer products.
 Who knows? If that happens, it's going to be good for all of us. And if they stay in
 the big chip server world, I think it's still good for us because all of the cloud compute
 might get cheaper because there's just more competition. Speaking of cheap synced rights,
 Microsoft India proposes Varuna, a scalable and low cost training of massive deep learning
 model system. So this is essentially an engineering paper that details how you can train big models
 on cheap and unreliable hardware. So the system uses both data parallelism as well as model
 pipelining. So you split up your data batches across different machines, you also split
 up your models across different machines. And if you do that in a smart way, you can
 achieve actual big throughput. So usually big models have to be trained on what they
 call hyper clusters, which means clusters that have very fast interconnect because in
 order to do something like an all reduce if you have to do layer normalization or batch
 normalization, I don't remember which one it is, sometimes you need to send data around,
 sometimes you need to send gradients around, and that costs a lot of compute and bandwidth
 and so on. So it's very interesting to see that these researchers are able to compete
 with these big hyper cluster training procedures and essentially bring that down to a heterogeneous
 clusters of spot instances that can die at any time. It's cool to see that AI training
 of these big models becomes something like a Kubernetes cluster where you can just add
 machines and the system will reconfigure itself to make optimal use of the machines however
 fast they may be connected and however long they might be up. So if you're looking for
 a cheap way to train a 200 billion parameter model, then this might be the way to go. Okay,
 here is a shout out to a few places. So the first shout out is to Laura Ruiz, his website
 where she replicates a bunch of things in young Lacan's and others papers called learning
 in high dimension always amounts to extrapolation. It's a very technical paper and Laura does
 a great job here, not only replicating the experiments in here, but providing really
 nice background and reasons and also the code that she uses to do everything. So I just
 thought this was really neat interleaving plots, code, math, and so on and really going
 through all of this. And in the end, actually being able to reproduce the plots of the papers,
 Yippee, there it is so beautiful, very reproduced much similar. If you want to follow Laura,
 definitely check out our website or GitHub. This is absolutely beautiful photo Laura.
 Good job. Right, another cool project is real life punch out by Ian Charnas. This is a really
 well made video about using body tracking models and pairing them up with punch out
 the N64 game. So you can actually play this in the browser, it tracks your arms, and you
 can punch using various boxing moves and play punch out. Not only that, but Ian actually
 went ahead and bought many cartridges of the game, as you can see in the background right
 here. And if you play it in the browser, it will actually use one of those cartridges
 because using just a ROM downloaded from the internet would violate the licensing agreements.
 So every game you play is essentially corresponding to a real life cartridge. As I said, the video
 is done extremely well. It's a fun video to watch. Or if you simply want to try it out,
 you can go to Ian's website and just play it by yourself. Nothing to install runs in
 the browser. Excellent. Alright, so this is the section where I provide some helpful things.
 First helpful thing market tech post writes Google AI introduces go emotions and NLP data
 set for fine grained emotion classification. I've actually shown this in last week's weights
 and biases ad if you have followed the weights and biases ads. But this is a data set where
 Reddit comments are annotated with one of I believe 28 different emotions contained in
 the comments. It's not only one emotion per comment, but technically any emotion could
 or could not appear in any comment. In total, there are 58,000 Reddit comments classified
 into on its 27 emotion categories 12 positive 11 negative four ambiguous and one neutral
 with that adds up to 28. I was right. So the data set creation process detailed here is
 detailing how they went about it, how they went about balancing the data, paying attention
 to the fact that Reddit isn't exactly a good replica of the entire world and so on. If
 you're interested, you can give this article a read, you can also look at the paper that
 goes along with the data set and you can use the data set if you want to try out your hand
 at emotion detection. I have to say it's gotten a bit tired to see NLP tutorials always doing
 sort of semantic classification where it's just positive or negative and this might just
 provide a little bit of a more challenging task here has this language interpretability
 tool it's open source and it's for visualizing and understanding NLP models. This provides
 various things you can look at embedding spaces of NLP tasks, it can analyze things like classification,
 regression, looking at attention heads, analyzing parts of the input, which parts are important
 for which things and so on. All in all, it's quite a rich tool and I encourage you to check
 it out if you're into language interpretability. Or if you want to just check out how your
 models do the things they're doing code is available tool is available. Okay, last week,
 we've reported on a rudali the Russian Dalí model. And now apparently the large model
 is available for download as one Reddit comment says, or much rather the edit of the comment
 says that the availability is on December 1. So expect that soon. machine on Twitter
 says after a year in dev, I'm happy to release the core of my Vtuber apps. Now Vtubers are
 special sort of things that I have never really touched on. But this seems to be a large community
 that transforms their body movements onto digital anime avatars, as you can see right
 here. So this also uses body pose tracking and apparently also face tracking in order
 to make your avatar do as you're doing code is available. And it's not only sort of for
 face and upper body, but you can also track your entire body movements and map them onto
 characters as you can see right here, it can do facial point tracking such that it really
 replicates your facial expressions. So there's never been a better time to become a Vtuber.
 Check out Khalid o kit on GitHub if you're interested. There's an article by Newsfile
 Corporation on Yahoo Finance that writes that artificial intelligence now makes it possible
 for investors to find promising new hidden gem meme tokens automatically. This isn't
 necessarily what you think you think while there's a company that tells me which meme
 tokens are good so I can buy it. No, no, no, no, no, no, no. See, this is an actual token
 itself. So you put money into the token, and then the token selects projects in which the
 money is to be invested. These projects it says are automatically selected using a special
 AI based sniper bot. So the AI will look at all the meme tokens, the dodge and the Shiba
 enu and the squid game tokens, and it will predict which ones will go up and then it
 will take all the money that is invested into the Finu token, put it into those tokens and
 then pay out the winnings to the holders of the Finu token. I mean, look at this for an
 enhanced version of this graphic, please. Yes, I want an enhanced version. Oh, wow,
 that's enhanced. That that is that is so hands. Absolutely. Currently, there is a website
 for this and it says vote for Finu help the price pump and hit the back there is a dodge.
 Okay people who want to make a quick buck using meme tokens that have absolutely no
 value whatsoever, are encouraged to buy a meme token. Excellent. Now I'm not saying
 this can't be done. Mean tokens are essentially like fashion that there's no reason why this
 particular that particular fashion should be in or out next year and yet it still happens
 and there might be ways to predict it. But still, whether or not this is the way to go
 can't tell. So I've mentioned this shoe does not exist last week. But there's also this
 sneaker does not exist. Look at that. And this is pretty cool. So this is a grid of
 AI generated sneakers, you can click on one, right, and then you can apparently edit that
 sneaker. So you can go normal to futuristic, you can go high creativity, that's very creative.
 You can change up the colors a little bit. Very cool, very functional. Look at that one.
 Yeah, futuristic, creative, light color. I mean, it's not super futuristic. But yeah,
 so shout out to this sneaker does not exist.com. Check it out. And that was already it for
 this week's ML news. I hope you had fun hit subscribe if you liked it. We're only 105,900,000
 subscribers behind PewDiePie. We can totally catch them. If we really do our jobs, tell
 three people they're going to tell three people is going to be fine. See you next Monday.
 Bye bye.