[ML News] DeepMind builds Gopher | Google builds GLaM | Suicide capsule uses AI to check access
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deepmind gopher retro toxicity ethical machine learning ethics ai ethics retrofit retrofit model retro transformer deepmind gopher google glam glam model glam transformer sparse transformer mixture of experts suicide capsule ai suicide ml news mlnews machine learning news kilcher news huggingface hugging face code parrot synthesia synthesia avatar
#mlnews #gopher #glam

Your updates on everything going on in the Machine Learning world.

Sponsor: Weights & Biases
https://wandb.me/yannic

OUTLINE:
0:00 - Intro & Overview
0:20 - Sponsor: Weights & Biases
3:05 - DeepMind releases 3 papers on large language models
11:45 - Hugging Face Blog: Training CodeParrot from scratch
14:25 - Paper: Pre-Training vision systems with noise
15:45 - DeepMind advances Quantum Mechanics
16:45 - GoogleAI trains GLaM: 1 Trillion Parameters Mixture of Experts Model
18:45 - Colin Raffel calls for building ML models like we build Open-Source software
22:05 - A rebuke of the hype around DeepMind's math paper
24:45 - Helpful Things
32:25 - Suicide Capsule plans AI to assess your mental state before use
35:15 - Synthesia raises 50M to develop AI avatars


Weights & Biases Embedding Projector
https://twitter.com/_ScottCondron/status/1469411468139536385?utm_source=pocket_mylist
https://docs.wandb.ai/ref/app/features/panels/weave/embedding-projector
https://wandb.ai/timssweeney/toy_datasets/reports/Feature-Report-W-B-Embeddings-Projector--VmlldzoxMjg2MjY4?accessToken=bo36zrgl0gref1th5nj59nrft9rc4r71s53zr2qvqlz68jwn8d8yyjdz73cqfyhq

DeepMind releases 3 papers on large language models
https://deepmind.com/blog/article/language-modelling-at-scale
https://arxiv.org/pdf/2112.04426.pdf
https://kstatic.googleusercontent.com/files/b068c6c0e64d6f933068f7de30ea722359ef87c6c14d3065856b86d44fbdf2dea3ff373ed9eb751514f242d20df9d6a468622fad093f962563545e7d0cdb9dba
https://arxiv.org/pdf/2112.04359.pdf
https://deepmind.com/research/publications/2021/improving-language-models-by-retrieving-from-trillions-of-tokens

Hugging Face Blog: Training CodeParrot from scratch
https://huggingface.co/blog/codeparrot?utm_source=pocket_mylist

Paper: Pre-Training vision systems with noise
https://mbaradad.github.io/learning_with_noise/

DeepMind advances Quantum Mechanics
https://deepmind.com/blog/article/Simulating-matter-on-the-quantum-scale-with-AI
https://storage.googleapis.com/deepmind-media/papers/Data_Driven_Density_Functional_Design/data_driven_density_functional_design_unformatted.pdf
https://github.com/deepmind/deepmind-research/tree/master/density_functional_approximation_dm21

GoogleAI trains GLaM: 1 Trillion Parameters Mixture of Experts Model
https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html

Colin Raffel calls for building ML models like we build Open-Source software
https://colinraffel.com/blog/a-call-to-build-models-like-we-build-open-source-software.html

A rebuke of the hype around DeepMind's math paper
https://arxiv.org/abs/2112.04324?s=09

Helpful Things
https://twitter.com/huggingface/status/1468996110207401992
https://docs.cohere.ai/prompt-engineering-wiki/?utm_source=pocket_mylist
https://github.blog/2021-12-08-improving-github-code-search/
https://huggingface.co/blog/data-measurements-tool
https://huggingface.co/spaces/huggingface/data-measurements-tool
https://blogs.microsoft.com/ai-for-business/building-ai-responsibly-from-research-to-practice/
https://techcommunity.microsoft.com/t5/azure-ai-blog/responsible-ai-dashboard-a-one-stop-shop-for-operationalizing/ba-p/3030944
https://github.com/minitorch/minitorch?utm_source=pocket_mylist
https://minitorch.github.io/
https://pandastutor.com/
https://pandastutor.com/vis.html
https://github.com/IAmPara0x/yuno
https://colab.research.google.com/drive/1WAewYgHDmDEWhPBBOvGgyLTiOaasVyOz?usp=sharing#scrollTo=hZamByTeBv3G
https://www.reddit.com/r/MachineLearning/comments/rbue4h/n_us_gov_launches_ml_competition_to_predict_snow/
https://www.drivendata.org/competitions/86/competition-reclamation-snow-water-dev/
https://www.reddit.com/r/MachineLearning/comments/rdb1uw/p_utttai_alphazerolike_solution_for_playing/
https://www.uttt.ai/
https://arxiv.org/abs/2112.02721?utm_source=pocket_mylist
https://arxiv.org/pdf/2112.02721.pdf
https://github.com/GEM-benchmark/NL-Augmenter
https://www.reddit.com/r/MachineLearning/comments/rdfdcv/p_collection_of_33_psychology_related_datasets/?utm_source=pocket_mylist

Suicide Capsule plans AI to assess your mental state before use
https://www.swissinfo.ch/eng/sci-tech/sarco-suicide-capsule--passes-legal-review--in-switzerland/46966510

Synthesia raises 50M to develop AI avatars
https://techcrunch.com/2021/12/08/synthesia-raises-50m-to-leverage-synthetic-avatars-for-corporate-training-and-more/
https://www.synthesia.io/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF

SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 DeepMind builds a dense language model with 280 billion parameters. Google builds a sparse language model with over a trillion parameters. And Microsoft has a new dashboard. Welcome to ML News. Hey there, this video is sponsored by Weights and Biases. Me and Weights and Biases, we've decided to take the next step in our relationship. And that means I now have my custom link, 1db.me slash Yannick. For all your needs, I actually don't know what's behind, I'm gonna look it up after. But there might be a surprise. Who knows what's behind that link? The only way you're gonna find out is by going to it. Anyway, today I want to tell you about a new feature in Weights and Biases. So I've previously told you about tables. Tables is this very cool thing in Weights and Biases that allows you to analyze your data, your models, your results, your outputs in a table form. But the table is like interactive. So the table can do anything from filter and group to display your plots, play little sound files, play GIFs and so on. And it's just an awesome way to look at your data from different angles. They have now added a new feature to tables called the embedding projector. So whenever I wanted to look at some sort of projection of my embeddings or data, I had to do that within the experiment and then log that like as a picture to TensorBoard. Now TensorBoard has also gained some projector view. But this here is really cool. So you can take any table and any columns of those tables as long as they're ints or floats. And you can use these projections to map them to a two dimensional space and then look at them in 2D. Now for that you have several algorithms at your disposal. On the left you can see a PCA projection of the digits data set and hovering over any given sample shows you more information. In this case, the sample itself. In the middle, you see a U map and on the right is a t-sne. You can interactively configure these projections, including their parameters, which columns are included, how the data is constructed and much, much more. And these are interactive, like you can do anything here that you would do in a regular interactive plot. And as always, you can then pull those into reports and show them together with data or with some explanation. And this is just a really cool tool to do data exploration or exploration of the predictions of your model. You can see you have all the power available here of regular weights and biases plots such as color coding, or intensity coding, whatever you want. Look at that. Isn't that a data set? Oh t-sne, what are you doing? Now I absolutely invite you to go check out weights and biases not only for the embedding projector, but as you know, they have tons and tons of features for both practitioners and researchers. It's completely free for personal use and academic use and no excuse not to try it. Thanks again to weights and biases for sponsoring this video. And let's get into it. DeepMind releases a blog post called language modeling at scale go for ethical considerations and retrieval that details not one but three new papers out of DeepMind. Paper is a huge language model and its biggest configuration is over 280 billion parameters. That is almost twice the size of GPT-3. Now the authors here evaluate the model on 152 diverse tasks and they achieve state of the art performance in the majority of them. The paper as you can see is pretty long as it needs its own table of contents, but it's essentially a big investigation into what these language models can do, what they cannot do and how they perform in the individual tasks. The main interest here is what happens if you scale these models up? What can you do and what can't you do? And the authors notes gains from scale are largest in areas such as reading comprehension, fact checking and the identification of toxic language, but logical and mathematical reasoning see less benefit. In order to train gopher, they also collect a new data set which they call massive text. It's a collection of large English language text data sets from multiple sources, web pages, books, news articles and code. So not only do the authors confirm that more text is a good thing, but they also confirm in their studies in their analysis that very much the quality of the input text is just as important as the amount of input text. So cleaning the data and also sampling the data according to its quality makes a big difference in these models. The authors note we provide a holistic analysis of the training data set and the models behavior covering the intersection of model scale with bias and toxicity. Now I have to say something like bias and toxicity is given a pretty big weight in this paper. I don't know why because it's an investigation into many, many things of these large language models. And I personally don't see bias and toxicity being like a specifically bad problem that specifically needs to be highlighted. It's not like we don't have enough problems on our hands with the 151 other problems. But for some reason, DeepMind chooses to highlight this one. The blog post also briefly goes into the main results, which were already mentioned in this short summary. But as you can see right here, gopher often beats GPT-3. However, it's still behind human experts in most tasks. And when it comes to things like scientific and mathematical reasoning, it actually just as GPT-3 does performs pretty poorly and purpose built systems to do mathematical reasoning, even though they are still lagging behind human experts are much better than something like gopher or GPT-3. I think this is to be expected as just sort of picking up from language, you learn a lot of things like a lot of factual knowledge about the world and a lot of things that people say and stories they tell and so on. Yet for something like mathematical reasoning, it is not as much a language input thing. It is much more an algorithm that you have to sort of practice over and over and someone needs to show you how to do it and specifically essentially program your brain to do an algorithm. Now I do believe there's evidence that large language models in principle can do these things. But what I'm saying is that if you simply feed a large language model, a lot of data from the internet is going to pick up on like common sense facts a lot more easily than on mathematical reasoning because I doubt there's many websites that say, you know, look here is how you do step by step logical inference. So the model essentially would have to pick it up through what amounts to reinforcement learning whereas common facts about the world, they can just recite from some website. So is it the lack of appropriate training data or is the model architecture simply incapable of performing logical reasoning? I believe the community is quite split on this point and it would be interesting to hear what you think. The second paper is called Ethical and Social Risks of Harm from Language Models and is an investigation a bit of a survey of different areas of risk about these language models. The abstract says the paper outlines six specific risk areas discrimination, exclusion and toxicity, information hazards, misinformation harms, malicious uses, human computer interaction harms and automation access and environmental harms. The most interesting paper though is the last paper it's called Improving Language Models by Retrieving from Trillions of Tokens. There is a special blog post to go along with the paper if you want a shorter more condensed version. But in essence, this is a language model. It's called retro that not only does it produce language, but as it produces language, it is able to go to a database of things that it can retrieve. So in this database, you can put all of Wikipedia here, they say GitHub, books, news and so on. Essentially, whatever you would usually train on. So your training corpus, you also make it indexable via a lookup index, then as you train your language model in each step of producing the next token, what you do is you take the current input or whatever you've produced so far, you go to that database, you retrieve the nearest neighbors of whatever your input is so far. And these nearest neighbors you retrieve with something like pre trained BERT embedding model, I guess you could also do some TF IDF things. So you want to get the sort of closest neighbors out of the training data set or the whatever database you have, and then you provide those to the language model as additional reference to take from the paper introduces a special chunked attention model such that it can actually refer to these individual passages that the retrieval step takes out without having the quadratic memory blow up of attention. And as you can see, it interleaves self attention layers like in a regular transformer language model with these cross attention layers that now attend to the retrieve things from the database. The result is pretty astounding. As they say, they can achieve sort of the performance of these large language models while having much, much less parameters. And it seems what's happening here is that we always used to think that for these large language models, you had to scale the data up so they know more stuff or can do more things. But in concordance with scaling up the data, you also had to scale up the model because what we do during training is kind of we take the data and we sort of embed the data into the weights of this neural network by training it read the reason GPT three knows so much is because we've baked all of this knowledge into the weight somewhere. So GPT three not only has the rules of how to produce language, but also sort of the knowledge that it will produce all in its weights. So we always used to scale data and model size and compute at the same time. Now it seems possible and that's what this research shows that you can in fact, take some of that data and sort of decouple it from the model size and the compute that you put in by supplying it at essentially inference time. So now the language model can be much more focused on how do I need to construct language it may have a little bit of knowledge in there, but it can always look up more knowledge at inference time and use sort of that to produce the output. The paper goes into more details about the architecture, the chunked attention mechanism and much more stuff. But what's also pretty cool is that you can if you want take just this transformer this language model and use it as a regular language model by not retrieving anything and that seems to work okay ish. So even if the model cannot retrieve something, it's still able to give good outputs not perfect, not the best but good. And conversely, it also seems to be quite easy to take a pre-trained language model and augment it by such a retrieval mechanism. So to what they call retrofit it, which is a wordplay because their models called retro. So this is like this is like a dad joke that's been in the making for you know, nine months or so. So I hope I hope you enjoy this moment where you can say, look, we retrofit the model. But it is pretty cool though, you can take a language model that's been pre-trained and with a bit of fine tuning, it seems you can make it use this retrieval mechanism and therefore you can supply it with much more data that has been trained on. This can also be a method to keep these models up to date because you know, the training data set gets older by the day, by definition and instead of retraining, you might be able in the future to just switch out the retrieval database and therefore keep the models outputs up to date. So it's been all pretty cool, if you are interested, check out the blog post, the papers and DeepMind, no affiliation. Leandro von Vera has a blog post on the hugging face blog called training code parrot from scratch where he goes in detail in through how you can train your own model that is like GitHub's copilot. So it takes your code and it suggests what next code you want to write. Now copilot by itself is an amazing system. And obviously there's there's a lot of engineering behind it, there is way more parameters than you could ever train. But if you want to train a small model from scratch or from a checkpoint, this is an excellent insight into how this is done. So it goes through everything getting the data, cleaning the data, training a tokenizer for code, actually training the model, evaluating it and everything. It shows you how to do some optimizations, like how you can make everything a bit more efficient by concatenating different samples. So you always fill out the context shows you what you need to pay attention to when cleaning the data set turns out on GitHub, very, very many files are actually duplicated. And that really hurts training performance goes through hyper parameters, it goes through data parallelism and optimizing your training code. And it's just super detailed. So here you can see, for example, the comparison of the accuracies and the code pair of models, even though they're quite small, they do actually get some significant ish performance. Now it's nowhere near open AI codecs model, which is the model powering GitHub copilot supposedly, but it still, you know, does something and that's pretty cool. So here you can see an example of this. So the prompt is a function definition called is even that returns true if a value is an even number, and then the model is asked to set up a unit test for is even. And as you can see right here, the completion that is given not only is it the correct name has a good doc string, but also it actually tests the function in question. And it doesn't really, you know, get what it's supposed to do. But still, the structure is sort of already there. So you could, you know, just assert like false right here. But as we know, these models really shine when it comes to like knowing how to handle API's of some libraries and so on, because supposedly, these libraries either themselves are on GitHub, or there are many code projects that already use these libraries. So the models would essentially know how to use the libraries and what functions to call and so on. Here you can see that the model is perfectly able to build a bird classifier. I guess, you know, this is also a bit of a shill for hugging face because it just takes two lines of code with their code base, but still models pretty cool. So if you're interested, definitely give this blog post a read. There's a paper out of MIT called learning to see by looking at noise. And this paper questions the paradigm of pre training on data by switching to pre training on noise, and they actually get some pretty decent results. They do investigate different styles of noise, so there is procedurally generated noise, statistical noise, there is initialized style, and so non trained style gowns, where you simply forward pass data and what comes out, you take as training images. And there is also feature visualization procedures of trained models. Now here you can see in dark the actual pre trained models on real images. And you can see that the models that have been pre trained on noise aren't that far behind. Especially interesting is that style gown models just initialized randomly and then forward propagated give pretty decent results. Now these results are on pre training on a data set, and then linearly adapting these models to image net, which is obviously not the most performant thing to do, but it gives sort of a baseline. Also interesting is that apparently Minecraft images also do quite well. There's much more to this paper, including feature visualizations, evaluations, and so on. If you're interested, paper code and data sets are available. DeepMind has another blog post called simulating matter on the quantum scale with AI. Now I have tried reading through this paper and even through the blog post. And honestly, I have no clue of anything quantum like quantum chemistry, anything like this. This is just beyond me. But this paper deals with the prediction of where electrons are in a molecule. So it turns out you don't actually need to track the individual electrons, you just sort of need to track the density function of where any electron could be at any time. And in order to predict that various approximations and heuristics are used, and turns out that if you use machine learning and a little bit of very clever data engineering and feature engineering, then you can come up with a system that outperforms any of these previous systems. Now, again, the paper has been published in science, I have no clue what any of this means. If you do, and if you're interested, go check it out. Google AI publishes a blog post called more efficient in context learning with glam. This goes along with a paper called glam efficient scaling of language models with mixture of experts. This is a model that is over a trillion parameters in size. Now, this is a sparse model. So it is not directly comparable to whatever the 175 billion parameters of GPT three, which is a dense model. So in a sparse model, what you do is that in the feed forward layer of the transformer layers, you would not activate all of the feed forward layer for every token, but you would route the tokens to one of many what are called experts. So these models are generally called mixture of expert models. So the idea is that you have this gating layer, and the gating layer decides which of the experts become activated. This results in each token only activating a small part of the network, which makes it way more energy efficient to actually forward propagate at inference time also makes it faster and with the current hardware and algorithm optimizations that the Google AI team has put in here, it does require more flops at training time because it trains on a way large or data set than current dense models. However, it does require actually less electricity. And that's pretty cool. I guess it's a little bit that you're trying to find some kind of a metric where you're better than anyone else. But I do find it cool that both at inference time and in terms of training energy consumed, this is actually the preferable model. Now, it is huge, and you need a huge architecture to train it. But I think that counts for all of the models currently, they do have a lot of investigations into comparing dense and sparse models. And they do generally find that the sparse models outperform the dense models given the same amount of training tokens and their final model outperforms GPT three on a number of natural language tasks. So pretty cool. If you're interested, check out the paper. Colin Raffel releases a call to build models like we build open source software. This is a blog post with a general appeal to the community where he first lists a bunch of the advantages of open source software versus closed source software and a bunch of features of open source development such as version control, submitting patches and pull requests, merging semantic versioning, compatibilities, and so on. And then he tries to make analogies to how we could develop models. So at the end, he has this paragraph right here where he details how a potential future could look. So this says researchers at Sullivan University decide to train a new language model called Clamp. They have limited access to computational resources. So they are only able to train the model for enough time to attain reasonable performance on a few downstream tasks after fine tuning, they set up a framework for testing the model's fine tuned performance on a suite of downstream tasks and release version 1.0.0 of the model to the world. Later, a different group of researchers at the University of Duxville make use of their computing cluster to perform additional training, use a training method that only updates a few of the model's parameters so that they can cheaply communicate the proposed changes back to Clamp's maintainers. The new model's performance is rapidly verified on the task suite thanks to the ability to reuse updates from previous fine tuning run. However, it turns out that the Fidmore Foundation has also been performing additional training in parallel. Fortunately, the updates by each organization can be merged and they are included in a new release of Clamp in version 1.0.1. And it goes on. So this tries to make a bunch of these analogies and I have to say some of them are pretty accurate and would be like nice to have, especially sort of this collaborative development of models, you release a checkpoint, someone else improves upon it, you sort of merge this together and so on, you raise like a pull request on a model. But some of these are a little bit more shady, like you would only update a small part of the model because that makes it cheap to communicate. Usually the communication overhead is in like distributed training where you need to communicate thousands and thousands of time. That's when it matters. But when I train a new model and I like raise a pull request, I don't think it matters whether I have 40 or 60 gigabytes of weights that I want to merge into the different model. Also sort of this notion of backwards compatibility, I think is a little different in real software versus versus models. And the only true example Colin gives here is that the model would still take the same inputs and give the same outputs. But that honestly that has nothing to do with machine learning. That is again, like that is a regress to actual software engineering, right? That would be using our old systems for software engineering. And in between somewhere is a model. So it might be a bit of a sort of forced analogy at some places. But I do think it's pretty cool. And I do think new paradigms of how we develop models together, especially as opposed to a few companies internally developing these huge models just in silos and then selling them via API's. But a few things are in the way, most notably the very, very often requirement to train things end to end, which sort of makes this whole, you know, modularity among models a bit tricky. If you want to read the whole blog post, feel free to check it out. Ernest David releases a paper on archive called deep learning and mathematical intuition, a review of Davies et al 2021. This is a response to DeepMinds paper about using deep learning in fundamental math. Now, ML News has reported on this with our outside reporter, Marcus Bedding last week. And this paper kind of criticizes the hype around this this math paper. Now, fair to say, this paper has been kind of overblown in pop culture, like, oh, AI solves math and whatnot. I mean, my own thumbnail was a clickbait for exactly this. But I just want to draw attention to the abstract here. In the not theory result, the role of deep learning was small and the conventional statistical analysis probably would have sufficed. In the representation theory result, the role of DL is much larger, however, is not very different in kind from what has been done in experimental mathematics for decades. Moreover, it is not clear whether the distinctive features of deep learning that make it useful here will apply across a wide range of mathematical problems. Finally, I argued that the deep learning here guides human intuition is unhelpful and misleading. What the deep learning does primarily does does primarily does is to mark many possible conjectures as false and a few others as possibly worthy of study. I don't think DeepMind has actually said anything else. Like just the amount of salt in this abstract is. I haven't actually read the paper, so the paper could be totally sane and reasonable. But the salt here is I can taste the salt through the internet. But I'm sorry, if a conventional statistical analysis would probably have sufficed, then why didn't you do a conventional statistical analysis? Why aren't you going out and doing conventional statistical analysis, getting more fundamental theorems or more results in mathematics? Why wouldn't that be like a better use of your time? No, I'm obviously like it is important to also criticize in academia. I think that that is a healthy part of the ecosystem. But let's be honest, this paper has mostly been overhyped by media and the paper itself has actually stated fairly accurately what the contribution of deep learning was. So I doubt that an academic paper is the correct refutation to media hype. I think that refutation has to actually just come from other media. But if you're interested in a more sober analysis, and maybe a little bit of salt, give this paper a read. Okay, some helpful things for this week. Transformers has a new release with lots of updates version 4.13.0 is out and has a lot of new models such as Segformer, ImageGPT, D'Berta v3 and the trainer now supports B Float 16 numbers. Excellent. So P or AI releases a really, really nice basic introduction to prompt engineering, where they show how to engineer prompts for very different tasks and what has generally worked in the past to give good outputs of these language models that you can query using in context learning. Check it out, they not only have posts on prompt engineering itself, but also how to handle temperature or how to set top K and top P variables and so on. Excellent. So it's a machine learning thing, but GitHub improves its code search. I have been previously not so happy with GitHub's code search, and they have a bunch of updates, a bunch of keywords, you can use a bunch of filters and regexes and so on. And I'm quite happy about that. So I thought I'd share it with you. Huggingface introduces the data measurements tool. It's an interactive toolkit for looking at data sets. This is a tool to do some basic investigation into data sets like show summary statistics drill down into some distributions like word count distributions, see if there's anything off if there's anything over or undersampled, look at associations between words and samples and so on. And the goal is, I think to also make this into a tool where you can create new data sets pretty easily. The data measurements tool like everything else is available on the hugging face hub as a space. Very similar Microsoft releases a responsible AI dashboard that has various tools to analyze the outputs of your models and whether or not they conform to some standards where the most mistakes are made and really drill down into performance issues. So here are a few things it supports error analysis, model interpretability, data explorer, model statistics, counterfactual analysis, causal inference, what if questions and more. This is important, especially for practitioners that are trying to actually build real products and need to diagnose various failure cases that might not necessarily be covered in the training data. Sasha Rush releases mini torch. This is a tutorial ish book ish thing, where he goes through a building torch from scratch or something like torch. So in this tutorial, you'll learn about mathematical operations, how you can build up a system that does auto differentiation, how you can build up a tensor class yourself, how you make everything more efficient and so on. And there is a GitHub repo to go along with this if you just want to skip to the end or if you want to follow along. Excellent. The pandas tutor is an introductory tool to pandas that lets you understand how pandas transforms your data. So in here, you'd put your pandas command your your Python code that operates on pandas data frames, and it would show you line by line what happens to your data. So here is a data set of dogs. If I go down, you can see it recognizes the first operation is filtering by a Boolean mask. And it shows me exactly what's happening in my data frame with a nice visualization and even a little bit of animation. The second line is a sort. So it shows me what thing it sorts by shows me where every data point is going, then there's a group by and finally a median which are visualized using colors. And again, a bunch of arrows, they do have more visualizations than just arrows and colors. But this is just an example. If you're new to pandas and try to understand what a given piece of code does or try to debug some kind of a bug that you have, this might be a nice place to look, you know, is a search engine that given a description gives you an appropriate anime to look at, I am not a big watcher of anime. But if you are, this might be just a tool for you though, if you are a big fan, you probably already know all of them. So you know, but it's a cool project. The author describes in detail in how this went about, there's a lot of analysis of the data set, the code is available, there's a collab where you can try it out. So here is an anime where the main character is very smart, but no one knows about it, you can set a slider for curiosity and you get various suggestions. The US Bureau of Reclamation has a competition where you have to predict how much water is released from snowpack. So this is a really important measurement because during the winter snow falls into the Rockies and then during the spring and summer it melts off and provides all the fresh water to essentially the western part of the US mainly and predicting where how much snow is and how much of it is going to melt is very crucial to planning ahead. There's actually $500,000 to win right here. This is split up so the overall winner gets 150k but if you are also the best in various regions, you can collect prize money from each of the regions. And there's also prize money for the best report. So yay. Internet user Arno Wachczynski writes the story about creating an Alpha Zero like solution for playing Ultimate Tic Tac Toe in the browser. This user did not know anything about web development when they started and it has resulted in a website where you can actually play this game. Now I didn't I didn't even know what this game was, but it's a very interesting game. So you play Tic Tac Toe, but it's it's sort of a super grid superimposed and your opponent will be able to play in the sub grid of sort of the cell you select right here. So if I select this cell, the opponent will be able to play in this cell the next move. So you kind of need to plan ahead and then if you win, let's just let's just screw up horribly right here. Let the opponent kind of win again in this cell, right? So if the opponent wins down there, then it's not over. But you sort of have to not only win the small games, you have to win like the super games. This, this is just for a human. This is crazy. And this user has developed a sort of an Alpha Zero like AI for this and the development is really nicely documented. So if you want to give it a try or if you want to follow sort of the development of this, check it out. NL Augmentor is a framework for task sensitive natural language augmentation. And as you can see, it has a bunch of authors. I'm reporting this because I've previously shouted out this project and I think it's a pretty cool initiative. The paper has collected augmentations, natural language augmentations from all users and anyone who submitted one is an author on the paper. Now whether authorship is meant for that, I don't know, but you know, if the foundation model team can do it, then certainly this is justified. The final library of NL Augmentor is available on GitHub and as far as I know, still being extended. Very cool. And lastly, there is a collection of 33 psychology related data sets user yumquair writes on Reddit. You can find the website open psychometrics and if you are interested in psychometrics and learning from that data, this might be just the opportunity for you. Swiss info writes sarco suicide capsule hopes to enter Switzerland. Now this seems horrifying by itself, but it was actually more horrifying. Initially, there is a long fact check along editorial note that the article was changed. It originally said this already passed legal review and that it works with various organizations within Switzerland, which is not the case. The capsule wants to enter the Swiss market and is currently in the process of entering the market. As you know, in Switzerland, assisted suicide by choice is legal and there are organizations that sort of consult with you and you have to justify to them why you want to go through with a suicide. Usually it's because you're terminally ill and you don't want to cause your family more trouble than needed. As far as I know, they do have a pretty high bar for when they will actually go through with the procedure. This company seeks to replace with the capsule. Here's a description. The person will get into the capsule and lie down is very comfortable. Oh, gee, thanks is very comfortable. They will be asked a number of questions and when they have answered, they may press the button inside the capsule, activating the mechanism in their own time. At that point, the oxygen will just be reduced and you'll fall asleep and die like I have no trouble with the method of dying, right? But they say our aim is to develop an artificial intelligence screening system to establish the person's mental capacity. Naturally, there is a lot of skepticism, especially on the part of psychiatrists. Yeah, you think but our original conceptual idea is that the person would do an online test and receive a code to access the sarco. Oh, wow. So right after I take the online test for what's your cheese type, I can also take the online test to get into the suicide machine. I mean, I have to say it is a tricky subject, right? Because you want to give people this opportunity. But also, if you think that there's an easy way to sort of assess consent and mental state, it is also big underestimation of how, for example, depression works and what it actually does to you and your mental state. So even though you might be sort of conscious and legally allowed to make decisions, it is still very, very tricky. Now I'm generally of the opinion that in principle, in principle, it might be possible that an AI system might be on par with a psychiatrist in assessing said mental state. But I don't think we're going to be there like right now or in the near future. But who knows? Maybe you'll end up in one of these pun intended. And lastly, TechCrunch writes Synthesia raises 50 million US dollars to leverage synthetic avatars for corporate training and more. Synthesia is a company that creates these virtual avatars. So here is the three step process, select your AI presenter, type in your script and get your video. Excellent. Now I'm absolutely for not actually needing to portray a human face anymore with this, like either you hire an actor or someone company internal needs to do it and their faces somewhere recorded and so on. So I can totally see why this is appealing. Ironically, the little chat that popped like who who who makes these chats who thinks these chats are a good idea. Like I've never ever ever entered anything into a chat that pops up on a website. Ironically, the person in the chat, as you can see, is one of the one of the avatars. So the company goes full meta right here in that the salesperson selling you the virtual avatars is a virtual salesperson. Excellent. Now of course, these virtual avatars are useful in certain situations, though it does seem a little bit dystopian. It also does seems that other industry, notably the adult industry might profit quite a bit more from them. But who knows, maybe there will be sort of a lashback and the desire for real humanity and actual imperfection and the most desirable actors will be ones with scars and no makeup and dirt and disformed faces and anything and everything that shows that they are not AI created, though I have my doubts about that. Alright, this was it for ML news. Thank you so much for listening, watching. Please check out weights and biases. Thank you so much for sponsoring this video and remember to keep your gradients low. Bye.
 DeepMind builds a dense language model with 280 billion parameters.
 Google builds a sparse language model with over a trillion parameters.
 And Microsoft has a new dashboard.
 Welcome to ML News.
 Hey there, this video is sponsored by Weights and Biases.
 Me and Weights and Biases, we've decided to take the next step in our relationship.
 And that means I now have my custom link, 1db.me slash Yannick.
 For all your needs, I actually don't know what's behind, I'm gonna look it up after.
 But there might be a surprise.
 Who knows what's behind that link?
 The only way you're gonna find out is by going to it.
 Anyway, today I want to tell you about a new feature in Weights and Biases.
 So I've previously told you about tables.
 Tables is this very cool thing in Weights and Biases that allows you to analyze your
 data, your models, your results, your outputs in a table form.
 But the table is like interactive.
 So the table can do anything from filter and group to display your plots, play little sound
 files, play GIFs and so on.
 And it's just an awesome way to look at your data from different angles.
 They have now added a new feature to tables called the embedding projector.
 So whenever I wanted to look at some sort of projection of my embeddings or data, I had
 to do that within the experiment and then log that like as a picture to TensorBoard.
 Now TensorBoard has also gained some projector view.
 But this here is really cool.
 So you can take any table and any columns of those tables as long as they're ints or
 floats.
 And you can use these projections to map them to a two dimensional space and then look at
 them in 2D.
 Now for that you have several algorithms at your disposal.
 On the left you can see a PCA projection of the digits data set and hovering over any
 given sample shows you more information.
 In this case, the sample itself.
 In the middle, you see a U map and on the right is a t-sne.
 You can interactively configure these projections, including their parameters, which columns
 are included, how the data is constructed and much, much more.
 And these are interactive, like you can do anything here that you would do in a regular
 interactive plot.
 And as always, you can then pull those into reports and show them together with data or
 with some explanation.
 And this is just a really cool tool to do data exploration or exploration of the predictions
 of your model.
 You can see you have all the power available here of regular weights and biases plots such
 as color coding, or intensity coding, whatever you want.
 Look at that.
 Isn't that a data set?
 Oh t-sne, what are you doing?
 Now I absolutely invite you to go check out weights and biases not only for the embedding
 projector, but as you know, they have tons and tons of features for both practitioners
 and researchers.
 It's completely free for personal use and academic use and no excuse not to try it.
 Thanks again to weights and biases for sponsoring this video.
 And let's get into it.
 DeepMind releases a blog post called language modeling at scale go for ethical considerations
 and retrieval that details not one but three new papers out of DeepMind.
 Paper is a huge language model and its biggest configuration is over 280 billion parameters.
 That is almost twice the size of GPT-3.
 Now the authors here evaluate the model on 152 diverse tasks and they achieve state of
 the art performance in the majority of them.
 The paper as you can see is pretty long as it needs its own table of contents, but it's
 essentially a big investigation into what these language models can do, what they cannot
 do and how they perform in the individual tasks.
 The main interest here is what happens if you scale these models up?
 What can you do and what can't you do?
 And the authors notes gains from scale are largest in areas such as reading comprehension,
 fact checking and the identification of toxic language, but logical and mathematical reasoning
 see less benefit.
 In order to train gopher, they also collect a new data set which they call massive text.
 It's a collection of large English language text data sets from multiple sources, web
 pages, books, news articles and code.
 So not only do the authors confirm that more text is a good thing, but they also confirm
 in their studies in their analysis that very much the quality of the input text is just
 as important as the amount of input text.
 So cleaning the data and also sampling the data according to its quality makes a big
 difference in these models.
 The authors note we provide a holistic analysis of the training data set and the models behavior
 covering the intersection of model scale with bias and toxicity.
 Now I have to say something like bias and toxicity is given a pretty big weight in this
 paper.
 I don't know why because it's an investigation into many, many things of these large language
 models.
 And I personally don't see bias and toxicity being like a specifically bad problem that
 specifically needs to be highlighted.
 It's not like we don't have enough problems on our hands with the 151 other problems.
 But for some reason, DeepMind chooses to highlight this one.
 The blog post also briefly goes into the main results, which were already mentioned in this
 short summary.
 But as you can see right here, gopher often beats GPT-3.
 However, it's still behind human experts in most tasks.
 And when it comes to things like scientific and mathematical reasoning, it actually just
 as GPT-3 does performs pretty poorly and purpose built systems to do mathematical reasoning,
 even though they are still lagging behind human experts are much better than something
 like gopher or GPT-3.
 I think this is to be expected as just sort of picking up from language, you learn a lot
 of things like a lot of factual knowledge about the world and a lot of things that people
 say and stories they tell and so on.
 Yet for something like mathematical reasoning, it is not as much a language input thing.
 It is much more an algorithm that you have to sort of practice over and over and someone
 needs to show you how to do it and specifically essentially program your brain to do an algorithm.
 Now I do believe there's evidence that large language models in principle can do these
 things.
 But what I'm saying is that if you simply feed a large language model, a lot of data
 from the internet is going to pick up on like common sense facts a lot more easily than
 on mathematical reasoning because I doubt there's many websites that say, you know,
 look here is how you do step by step logical inference.
 So the model essentially would have to pick it up through what amounts to reinforcement
 learning whereas common facts about the world, they can just recite from some website.
 So is it the lack of appropriate training data or is the model architecture simply incapable
 of performing logical reasoning?
 I believe the community is quite split on this point and it would be interesting to
 hear what you think.
 The second paper is called Ethical and Social Risks of Harm from Language Models and is
 an investigation a bit of a survey of different areas of risk about these language models.
 The abstract says the paper outlines six specific risk areas discrimination, exclusion and toxicity,
 information hazards, misinformation harms, malicious uses, human computer interaction
 harms and automation access and environmental harms.
 The most interesting paper though is the last paper it's called Improving Language Models
 by Retrieving from Trillions of Tokens.
 There is a special blog post to go along with the paper if you want a shorter more condensed
 version.
 But in essence, this is a language model.
 It's called retro that not only does it produce language, but as it produces language, it
 is able to go to a database of things that it can retrieve.
 So in this database, you can put all of Wikipedia here, they say GitHub, books, news and so
 on.
 Essentially, whatever you would usually train on.
 So your training corpus, you also make it indexable via a lookup index, then as you
 train your language model in each step of producing the next token, what you do is you
 take the current input or whatever you've produced so far, you go to that database,
 you retrieve the nearest neighbors of whatever your input is so far.
 And these nearest neighbors you retrieve with something like pre trained BERT embedding
 model, I guess you could also do some TF IDF things.
 So you want to get the sort of closest neighbors out of the training data set or the whatever
 database you have, and then you provide those to the language model as additional reference
 to take from the paper introduces a special chunked attention model such that it can actually
 refer to these individual passages that the retrieval step takes out without having the
 quadratic memory blow up of attention.
 And as you can see, it interleaves self attention layers like in a regular transformer language
 model with these cross attention layers that now attend to the retrieve things from the
 database.
 The result is pretty astounding.
 As they say, they can achieve sort of the performance of these large language models
 while having much, much less parameters.
 And it seems what's happening here is that we always used to think that for these large
 language models, you had to scale the data up so they know more stuff or can do more
 things.
 But in concordance with scaling up the data, you also had to scale up the model because
 what we do during training is kind of we take the data and we sort of embed the data into
 the weights of this neural network by training it read the reason GPT three knows so much
 is because we've baked all of this knowledge into the weight somewhere.
 So GPT three not only has the rules of how to produce language, but also sort of the
 knowledge that it will produce all in its weights.
 So we always used to scale data and model size and compute at the same time.
 Now it seems possible and that's what this research shows that you can in fact, take
 some of that data and sort of decouple it from the model size and the compute that you
 put in by supplying it at essentially inference time.
 So now the language model can be much more focused on how do I need to construct language
 it may have a little bit of knowledge in there, but it can always look up more knowledge at
 inference time and use sort of that to produce the output.
 The paper goes into more details about the architecture, the chunked attention mechanism
 and much more stuff.
 But what's also pretty cool is that you can if you want take just this transformer this
 language model and use it as a regular language model by not retrieving anything and that
 seems to work okay ish.
 So even if the model cannot retrieve something, it's still able to give good outputs not perfect,
 not the best but good.
 And conversely, it also seems to be quite easy to take a pre-trained language model
 and augment it by such a retrieval mechanism.
 So to what they call retrofit it, which is a wordplay because their models called retro.
 So this is like this is like a dad joke that's been in the making for you know, nine months
 or so.
 So I hope I hope you enjoy this moment where you can say, look, we retrofit the model.
 But it is pretty cool though, you can take a language model that's been pre-trained and
 with a bit of fine tuning, it seems you can make it use this retrieval mechanism and therefore
 you can supply it with much more data that has been trained on.
 This can also be a method to keep these models up to date because you know, the training
 data set gets older by the day, by definition and instead of retraining, you might be able
 in the future to just switch out the retrieval database and therefore keep the models outputs
 up to date.
 So it's been all pretty cool, if you are interested, check out the blog post, the papers and DeepMind,
 no affiliation.
 Leandro von Vera has a blog post on the hugging face blog called training code parrot from
 scratch where he goes in detail in through how you can train your own model that is like
 GitHub's copilot.
 So it takes your code and it suggests what next code you want to write.
 Now copilot by itself is an amazing system.
 And obviously there's there's a lot of engineering behind it, there is way more parameters than
 you could ever train.
 But if you want to train a small model from scratch or from a checkpoint, this is an excellent
 insight into how this is done.
 So it goes through everything getting the data, cleaning the data, training a tokenizer
 for code, actually training the model, evaluating it and everything.
 It shows you how to do some optimizations, like how you can make everything a bit more
 efficient by concatenating different samples.
 So you always fill out the context shows you what you need to pay attention to when cleaning
 the data set turns out on GitHub, very, very many files are actually duplicated.
 And that really hurts training performance goes through hyper parameters, it goes through
 data parallelism and optimizing your training code.
 And it's just super detailed.
 So here you can see, for example, the comparison of the accuracies and the code pair of models,
 even though they're quite small, they do actually get some significant ish performance.
 Now it's nowhere near open AI codecs model, which is the model powering GitHub copilot
 supposedly, but it still, you know, does something and that's pretty cool.
 So here you can see an example of this.
 So the prompt is a function definition called is even that returns true if a value is an
 even number, and then the model is asked to set up a unit test for is even.
 And as you can see right here, the completion that is given not only is it the correct name
 has a good doc string, but also it actually tests the function in question.
 And it doesn't really, you know, get what it's supposed to do.
 But still, the structure is sort of already there.
 So you could, you know, just assert like false right here.
 But as we know, these models really shine when it comes to like knowing how to handle
 API's of some libraries and so on, because supposedly, these libraries either themselves
 are on GitHub, or there are many code projects that already use these libraries.
 So the models would essentially know how to use the libraries and what functions to call
 and so on.
 Here you can see that the model is perfectly able to build a bird classifier.
 I guess, you know, this is also a bit of a shill for hugging face because it just takes
 two lines of code with their code base, but still models pretty cool.
 So if you're interested, definitely give this blog post a read.
 There's a paper out of MIT called learning to see by looking at noise.
 And this paper questions the paradigm of pre training on data by switching to pre training
 on noise, and they actually get some pretty decent results.
 They do investigate different styles of noise, so there is procedurally generated noise,
 statistical noise, there is initialized style, and so non trained style gowns, where you
 simply forward pass data and what comes out, you take as training images.
 And there is also feature visualization procedures of trained models.
 Now here you can see in dark the actual pre trained models on real images.
 And you can see that the models that have been pre trained on noise aren't that far
 behind.
 Especially interesting is that style gown models just initialized randomly and then
 forward propagated give pretty decent results.
 Now these results are on pre training on a data set, and then linearly adapting these
 models to image net, which is obviously not the most performant thing to do, but it gives
 sort of a baseline.
 Also interesting is that apparently Minecraft images also do quite well.
 There's much more to this paper, including feature visualizations, evaluations, and so
 on.
 If you're interested, paper code and data sets are available.
 DeepMind has another blog post called simulating matter on the quantum scale with AI.
 Now I have tried reading through this paper and even through the blog post.
 And honestly, I have no clue of anything quantum like quantum chemistry, anything like this.
 This is just beyond me.
 But this paper deals with the prediction of where electrons are in a molecule.
 So it turns out you don't actually need to track the individual electrons, you just sort
 of need to track the density function of where any electron could be at any time.
 And in order to predict that various approximations and heuristics are used, and turns out that
 if you use machine learning and a little bit of very clever data engineering and feature
 engineering, then you can come up with a system that outperforms any of these previous systems.
 Now, again, the paper has been published in science, I have no clue what any of this means.
 If you do, and if you're interested, go check it out.
 Google AI publishes a blog post called more efficient in context learning with glam.
 This goes along with a paper called glam efficient scaling of language models with mixture of
 experts.
 This is a model that is over a trillion parameters in size.
 Now, this is a sparse model.
 So it is not directly comparable to whatever the 175 billion parameters of GPT three, which
 is a dense model.
 So in a sparse model, what you do is that in the feed forward layer of the transformer
 layers, you would not activate all of the feed forward layer for every token, but you
 would route the tokens to one of many what are called experts.
 So these models are generally called mixture of expert models.
 So the idea is that you have this gating layer, and the gating layer decides which of the
 experts become activated.
 This results in each token only activating a small part of the network, which makes it
 way more energy efficient to actually forward propagate at inference time also makes it
 faster and with the current hardware and algorithm optimizations that the Google AI team has
 put in here, it does require more flops at training time because it trains on a way large
 or data set than current dense models.
 However, it does require actually less electricity.
 And that's pretty cool.
 I guess it's a little bit that you're trying to find some kind of a metric where you're
 better than anyone else.
 But I do find it cool that both at inference time and in terms of training energy consumed,
 this is actually the preferable model.
 Now, it is huge, and you need a huge architecture to train it.
 But I think that counts for all of the models currently, they do have a lot of investigations
 into comparing dense and sparse models.
 And they do generally find that the sparse models outperform the dense models given the
 same amount of training tokens and their final model outperforms GPT three on a number of
 natural language tasks.
 So pretty cool.
 If you're interested, check out the paper.
 Colin Raffel releases a call to build models like we build open source software.
 This is a blog post with a general appeal to the community where he first lists a bunch
 of the advantages of open source software versus closed source software and a bunch
 of features of open source development such as version control, submitting patches and
 pull requests, merging semantic versioning, compatibilities, and so on.
 And then he tries to make analogies to how we could develop models.
 So at the end, he has this paragraph right here where he details how a potential future
 could look.
 So this says researchers at Sullivan University decide to train a new language model called
 Clamp.
 They have limited access to computational resources.
 So they are only able to train the model for enough time to attain reasonable performance
 on a few downstream tasks after fine tuning, they set up a framework for testing the model's
 fine tuned performance on a suite of downstream tasks and release version 1.0.0 of the model
 to the world.
 Later, a different group of researchers at the University of Duxville make use of their
 computing cluster to perform additional training, use a training method that only updates a
 few of the model's parameters so that they can cheaply communicate the proposed changes
 back to Clamp's maintainers.
 The new model's performance is rapidly verified on the task suite thanks to the ability to
 reuse updates from previous fine tuning run.
 However, it turns out that the Fidmore Foundation has also been performing additional training
 in parallel.
 Fortunately, the updates by each organization can be merged and they are included in a new
 release of Clamp in version 1.0.1.
 And it goes on.
 So this tries to make a bunch of these analogies and I have to say some of them are pretty
 accurate and would be like nice to have, especially sort of this collaborative development of
 models, you release a checkpoint, someone else improves upon it, you sort of merge this together
 and so on, you raise like a pull request on a model.
 But some of these are a little bit more shady, like you would only update a small part of
 the model because that makes it cheap to communicate.
 Usually the communication overhead is in like distributed training where you need to communicate
 thousands and thousands of time.
 That's when it matters.
 But when I train a new model and I like raise a pull request, I don't think it matters whether
 I have 40 or 60 gigabytes of weights that I want to merge into the different model.
 Also sort of this notion of backwards compatibility, I think is a little different in real software
 versus versus models.
 And the only true example Colin gives here is that the model would still take the same
 inputs and give the same outputs.
 But that honestly that has nothing to do with machine learning.
 That is again, like that is a regress to actual software engineering, right?
 That would be using our old systems for software engineering.
 And in between somewhere is a model.
 So it might be a bit of a sort of forced analogy at some places.
 But I do think it's pretty cool.
 And I do think new paradigms of how we develop models together, especially as opposed to
 a few companies internally developing these huge models just in silos and then selling
 them via API's.
 But a few things are in the way, most notably the very, very often requirement to train
 things end to end, which sort of makes this whole, you know, modularity among models a
 bit tricky.
 If you want to read the whole blog post, feel free to check it out.
 Ernest David releases a paper on archive called deep learning and mathematical intuition,
 a review of Davies et al 2021.
 This is a response to DeepMinds paper about using deep learning in fundamental math.
 Now, ML News has reported on this with our outside reporter, Marcus Bedding last week.
 And this paper kind of criticizes the hype around this this math paper.
 Now, fair to say, this paper has been kind of overblown in pop culture, like, oh, AI
 solves math and whatnot.
 I mean, my own thumbnail was a clickbait for exactly this.
 But I just want to draw attention to the abstract here.
 In the not theory result, the role of deep learning was small and the conventional statistical
 analysis probably would have sufficed.
 In the representation theory result, the role of DL is much larger, however, is not very
 different in kind from what has been done in experimental mathematics for decades.
 Moreover, it is not clear whether the distinctive features of deep learning that make it useful
 here will apply across a wide range of mathematical problems.
 Finally, I argued that the deep learning here guides human intuition is unhelpful and misleading.
 What the deep learning does primarily does does primarily does is to mark many possible
 conjectures as false and a few others as possibly worthy of study.
 I don't think DeepMind has actually said anything else.
 Like just the amount of salt in this abstract is.
 I haven't actually read the paper, so the paper could be totally sane and reasonable.
 But the salt here is I can taste the salt through the internet.
 But I'm sorry, if a conventional statistical analysis would probably have sufficed, then
 why didn't you do a conventional statistical analysis?
 Why aren't you going out and doing conventional statistical analysis, getting more fundamental
 theorems or more results in mathematics?
 Why wouldn't that be like a better use of your time?
 No, I'm obviously like it is important to also criticize in academia.
 I think that that is a healthy part of the ecosystem.
 But let's be honest, this paper has mostly been overhyped by media and the paper itself
 has actually stated fairly accurately what the contribution of deep learning was.
 So I doubt that an academic paper is the correct refutation to media hype.
 I think that refutation has to actually just come from other media.
 But if you're interested in a more sober analysis, and maybe a little bit of salt, give this
 paper a read.
 Okay, some helpful things for this week.
 Transformers has a new release with lots of updates version 4.13.0 is out and has a lot
 of new models such as Segformer, ImageGPT, D'Berta v3 and the trainer now supports B
 Float 16 numbers.
 Excellent.
 So P or AI releases a really, really nice basic introduction to prompt engineering,
 where they show how to engineer prompts for very different tasks and what has generally
 worked in the past to give good outputs of these language models that you can query using
 in context learning.
 Check it out, they not only have posts on prompt engineering itself, but also how to
 handle temperature or how to set top K and top P variables and so on.
 Excellent.
 So it's a machine learning thing, but GitHub improves its code search.
 I have been previously not so happy with GitHub's code search, and they have a bunch of updates,
 a bunch of keywords, you can use a bunch of filters and regexes and so on.
 And I'm quite happy about that.
 So I thought I'd share it with you.
 Huggingface introduces the data measurements tool.
 It's an interactive toolkit for looking at data sets.
 This is a tool to do some basic investigation into data sets like show summary statistics
 drill down into some distributions like word count distributions, see if there's anything
 off if there's anything over or undersampled, look at associations between words and samples
 and so on.
 And the goal is, I think to also make this into a tool where you can create new data
 sets pretty easily.
 The data measurements tool like everything else is available on the hugging face hub
 as a space.
 Very similar Microsoft releases a responsible AI dashboard that has various tools to analyze
 the outputs of your models and whether or not they conform to some standards where the
 most mistakes are made and really drill down into performance issues.
 So here are a few things it supports error analysis, model interpretability, data explorer,
 model statistics, counterfactual analysis, causal inference, what if questions and more.
 This is important, especially for practitioners that are trying to actually build real products
 and need to diagnose various failure cases that might not necessarily be covered in the
 training data.
 Sasha Rush releases mini torch.
 This is a tutorial ish book ish thing, where he goes through a building torch from scratch
 or something like torch.
 So in this tutorial, you'll learn about mathematical operations, how you can build up a system
 that does auto differentiation, how you can build up a tensor class yourself, how you
 make everything more efficient and so on.
 And there is a GitHub repo to go along with this if you just want to skip to the end or
 if you want to follow along.
 Excellent.
 The pandas tutor is an introductory tool to pandas that lets you understand how pandas
 transforms your data.
 So in here, you'd put your pandas command your your Python code that operates on pandas
 data frames, and it would show you line by line what happens to your data.
 So here is a data set of dogs.
 If I go down, you can see it recognizes the first operation is filtering by a Boolean
 mask.
 And it shows me exactly what's happening in my data frame with a nice visualization and
 even a little bit of animation.
 The second line is a sort.
 So it shows me what thing it sorts by shows me where every data point is going, then there's
 a group by and finally a median which are visualized using colors.
 And again, a bunch of arrows, they do have more visualizations than just arrows and colors.
 But this is just an example.
 If you're new to pandas and try to understand what a given piece of code does or try to
 debug some kind of a bug that you have, this might be a nice place to look, you know, is
 a search engine that given a description gives you an appropriate anime to look at, I am
 not a big watcher of anime.
 But if you are, this might be just a tool for you though, if you are a big fan, you
 probably already know all of them.
 So you know, but it's a cool project.
 The author describes in detail in how this went about, there's a lot of analysis of the
 data set, the code is available, there's a collab where you can try it out.
 So here is an anime where the main character is very smart, but no one knows about it,
 you can set a slider for curiosity and you get various suggestions.
 The US Bureau of Reclamation has a competition where you have to predict how much water is
 released from snowpack.
 So this is a really important measurement because during the winter snow falls into
 the Rockies and then during the spring and summer it melts off and provides all the fresh
 water to essentially the western part of the US mainly and predicting where how much snow
 is and how much of it is going to melt is very crucial to planning ahead.
 There's actually $500,000 to win right here.
 This is split up so the overall winner gets 150k but if you are also the best in various
 regions, you can collect prize money from each of the regions.
 And there's also prize money for the best report.
 So yay.
 Internet user Arno Wachczynski writes the story about creating an Alpha Zero like solution
 for playing Ultimate Tic Tac Toe in the browser.
 This user did not know anything about web development when they started and it has resulted
 in a website where you can actually play this game.
 Now I didn't I didn't even know what this game was, but it's a very interesting game.
 So you play Tic Tac Toe, but it's it's sort of a super grid superimposed and your opponent
 will be able to play in the sub grid of sort of the cell you select right here.
 So if I select this cell, the opponent will be able to play in this cell the next move.
 So you kind of need to plan ahead and then if you win, let's just let's just screw up
 horribly right here.
 Let the opponent kind of win again in this cell, right?
 So if the opponent wins down there, then it's not over.
 But you sort of have to not only win the small games, you have to win like the super games.
 This, this is just for a human.
 This is crazy.
 And this user has developed a sort of an Alpha Zero like AI for this and the development
 is really nicely documented.
 So if you want to give it a try or if you want to follow sort of the development of
 this, check it out.
 NL Augmentor is a framework for task sensitive natural language augmentation.
 And as you can see, it has a bunch of authors.
 I'm reporting this because I've previously shouted out this project and I think it's
 a pretty cool initiative.
 The paper has collected augmentations, natural language augmentations from all users and
 anyone who submitted one is an author on the paper.
 Now whether authorship is meant for that, I don't know, but you know, if the foundation
 model team can do it, then certainly this is justified.
 The final library of NL Augmentor is available on GitHub and as far as I know, still being
 extended.
 Very cool.
 And lastly, there is a collection of 33 psychology related data sets user yumquair writes on
 Reddit.
 You can find the website open psychometrics and if you are interested in psychometrics
 and learning from that data, this might be just the opportunity for you.
 Swiss info writes sarco suicide capsule hopes to enter Switzerland.
 Now this seems horrifying by itself, but it was actually more horrifying.
 Initially, there is a long fact check along editorial note that the article was changed.
 It originally said this already passed legal review and that it works with various organizations
 within Switzerland, which is not the case.
 The capsule wants to enter the Swiss market and is currently in the process of entering
 the market.
 As you know, in Switzerland, assisted suicide by choice is legal and there are organizations
 that sort of consult with you and you have to justify to them why you want to go through
 with a suicide.
 Usually it's because you're terminally ill and you don't want to cause your family more
 trouble than needed.
 As far as I know, they do have a pretty high bar for when they will actually go through
 with the procedure.
 This company seeks to replace with the capsule.
 Here's a description.
 The person will get into the capsule and lie down is very comfortable.
 Oh, gee, thanks is very comfortable.
 They will be asked a number of questions and when they have answered, they may press the
 button inside the capsule, activating the mechanism in their own time.
 At that point, the oxygen will just be reduced and you'll fall asleep and die like I have
 no trouble with the method of dying, right?
 But they say our aim is to develop an artificial intelligence screening system to establish
 the person's mental capacity.
 Naturally, there is a lot of skepticism, especially on the part of psychiatrists.
 Yeah, you think but our original conceptual idea is that the person would do an online
 test and receive a code to access the sarco.
 Oh, wow.
 So right after I take the online test for what's your cheese type, I can also take the
 online test to get into the suicide machine.
 I mean, I have to say it is a tricky subject, right?
 Because you want to give people this opportunity.
 But also, if you think that there's an easy way to sort of assess consent and mental state,
 it is also big underestimation of how, for example, depression works and what it actually
 does to you and your mental state.
 So even though you might be sort of conscious and legally allowed to make decisions, it
 is still very, very tricky.
 Now I'm generally of the opinion that in principle, in principle, it might be possible that an
 AI system might be on par with a psychiatrist in assessing said mental state.
 But I don't think we're going to be there like right now or in the near future.
 But who knows?
 Maybe you'll end up in one of these pun intended.
 And lastly, TechCrunch writes Synthesia raises 50 million US dollars to leverage synthetic
 avatars for corporate training and more.
 Synthesia is a company that creates these virtual avatars.
 So here is the three step process, select your AI presenter, type in your script and
 get your video.
 Excellent.
 Now I'm absolutely for not actually needing to portray a human face anymore with this,
 like either you hire an actor or someone company internal needs to do it and their faces somewhere
 recorded and so on.
 So I can totally see why this is appealing.
 Ironically, the little chat that popped like who who who makes these chats who thinks these
 chats are a good idea.
 Like I've never ever ever entered anything into a chat that pops up on a website.
 Ironically, the person in the chat, as you can see, is one of the one of the avatars.
 So the company goes full meta right here in that the salesperson selling you the virtual
 avatars is a virtual salesperson.
 Excellent.
 Now of course, these virtual avatars are useful in certain situations, though it does seem
 a little bit dystopian.
 It also does seems that other industry, notably the adult industry might profit quite a bit
 more from them.
 But who knows, maybe there will be sort of a lashback and the desire for real humanity
 and actual imperfection and the most desirable actors will be ones with scars and no makeup
 and dirt and disformed faces and anything and everything that shows that they are not
 AI created, though I have my doubts about that.
 Alright, this was it for ML news.
 Thank you so much for listening, watching.
 Please check out weights and biases.
 Thank you so much for sponsoring this video and remember to keep your gradients low.
 Bye.