[ML News] DeepMind tackles Math | Microsoft does more with less | Timnit Gebru launches DAIR
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deepmind ai math machine learning math deepmind math topology deepmind topology knot theory ai fundamental math deepmind representation theory deepmind mathematics gebru timnit gebru gebru dair timnit gebru research institute microsoft turing neurips neurips ethics review machine learning ethics helpful things sagemaker canvas rtx 3090 nvidia
#mlnews #deepmind #ai

The most trusted model in News!

Get started with Weights & Biases here: https://wandb.me/yannic
(it's free forever for personal use)

OUTLINE:
0:00 - Intro
0:15 - Sponsor: Weights & Biases
3:10 - DeepMind tackles fundamental math
6:45 - Microsoft focuses on scaling effectively and efficiently
10:15 - NeurIPS Anthology Visualization
13:30 - Timnit Gebru launches research institute independent from big tech
16:50 - SageMaker Canvas for no-code ML
17:50 - Help, Help!
21:40 - Cornelius Emde wins the 3090
21:55 - A retrospective on the NeurIPS 2021 ethics review process

References:
DeepMind tackles fundamental math
https://deepmind.com/blog/article/exploring-the-beauty-of-pure-mathematics-in-novel-ways?utm_source=pocket_mylist
https://www.nature.com/articles/s41586-021-04086-x?utm_source=pocket_mylist

Microsoft focuses on scaling effectively and efficiently
https://www.microsoft.com/en-us/research/blog/efficiently-and-effectively-scaling-up-language-model-pretraining-for-best-language-representation-model-on-glue-and-superglue/?OCID=msr_blog_TNLRV5_tw

NeurIPS Anthology Visualization
https://neuripsav.vizhub.ai/blog/
https://neuripsav.vizhub.ai/

Timnit Gebru launches research institute independent from big tech
https://www.washingtonpost.com/technology/2021/12/02/timnit-gebru-dair/
https://www.dair-institute.org/about
https://www.theguardian.com/commentisfree/2021/dec/06/google-silicon-valley-ai-timnit-gebru

SageMaker Canvas for no-code ML
https://aws.amazon.com/blogs/aws/announcing-amazon-sagemaker-canvas-a-visual-no-code-machine-learning-capability-for-business-analysts/

Help, Help!
https://macberth.netlify.app/
https://huggingface.co/emanjavacas/MacBERTh/tree/main
https://developer.nvidia.com/blog/nvidia-announces-tensorrt-8-2-and-integrations-with-pytorch-and-tensorflow/?ncid=so-twit-314589#cid=dl13_so-twit_en-us
https://opacus.ai/
https://twitter.com/naotokui_en/status/1466320722825920515
https://colab.research.google.com/drive/1H_g60Q_XELJ2VJu4GF7KY8111ce4VLwd?usp=sharing#scrollTo=JyNp3rwoWOQd
https://twitter.com/ThomasSimonini/status/1466437571303649301?utm_source=pocket_mylist
https://github.com/karpathy/arxiv-sanity-lite
https://arxiv-sanity-lite.com/
https://www.youtube.com/watch?v=01ENzpkjOCE
https://github.com/Felix-Petersen/algovision
https://github.com/rentruewang/koila?utm_source=pocket_mylist
https://github.com/YeWR/EfficientZero

Cornelius Emde wins the 3090
https://twitter.com/CorEmde/status/1466122212000374793

A retrospective on the NeurIPS 2021 ethics review process
https://blog.neurips.cc/2021/12/03/a-retrospective-on-the-neurips-2021-ethics-review-process/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 DeepMind tackles fundamental mathematics, Microsoft trains its most efficient and effective language model yet, and Timniggebru launches her own research institute. Welcome to ML News! Look at this, look at what I got as a Christmas present. It is a swag package from Weights and Biases. So, so if you look, there's lots of like yellow fuzzy fuzzy stuff to package, but mainly these are socks, Weights and Biases themed socks. Look at that. It's Weights and Biases socks. They have like little B's and little ones. Oh, I get it. Now you can see me here actually on camera realizing the following. See Weights and Biases URL is 1db.com. It's W and B. Now I have not realized this before, but the 1d and the B obviously stand for this URL. Now you can see me realize this right here on camera. Watch. It's 1db, like a 1d and a B. I just got this right, like literally I did not get this until right now. A 1d and a B. And then most importantly, this thing right here, which is a... Mug. Excellent. And this is really cool. Look at that. Like it's a colorless logo. It's kind of imprinted in metal. This is very cool cup. One sec. All right. I filled this up with tea. It is actually still steaming. It's completely hot on the inside, completely cool on the outside. Excellent. Thank you very much Weights and Biases for this awesome Christmas gift. Coincidentally, this video is sponsored by Weights and Biases. If you don't know Weights and Biases yet, please go check them out. Weights and Biases is the tool for your machine learning needs. It can do experiment tracking. One line of code tracks your experiments to the cloud. Nicely viewable. For every experiment, you can save all the output, all the logs, all the graphs. You can compare experiments. Weights and Biases can track your data sets and your models and save them as artifacts in the cloud. You'll know exactly how to reproduce every single thing there is. They have a really neat feature called tables where you can analyze your data, filter it, and really go into the depth of where your models still need improvement. This is not only useful during experimentation. It's actually useful all the way to deployment and monitoring after you've deployed your model. And then lastly, you can also pull all of this into reports, which is an interactive document that you can send to your boss, your team members, your clients even, and show them interactively how their stuff is doing. Reports are living documents with interactive plots and tables and all of the other features. So if you still do ML tooling by hand, give Weights and Biases a try. It's completely free for personal use and for academic use. They have solutions on cloud and on premise. There's no excuse not to check them out. Again, thank you so much, Weights and Biases, for sponsoring this video, for the awesome gift package. As you see, I am very bribable. And let's get into the video. DeepMind has a new blog post called Exploring the Beauty of Pure Mathematics in Novel Ways. And this blog post goes along with a paper in the journal Nature called Advancing Mathematics by Guiding Human Intuition with AI. This is a joint effort by DeepMind scholars and people in the actual mathematical fields to use AI to make new mathematical discoveries. Now, by new mathematical discoveries, I don't mean like the last digit of pi or something like this. These are actual fundamental theorems in fields like topology. Now, because I'm pretty bad at fundamental math, right now I'm actually going to speak to an outside correspondent who gives us the details on this story. I'm speaking live to Marcus Bedding. Marcus, it's very nice to have you on the show. Hi, Onik. Thanks for having me. Nice to be on the show. In fact, I'm standing in front of the building where math was once performed, apparently. So, Marcus, tell us, has DeepMind solved math? Is AI doing math now? Are mathematicians going to be obsolete? What's your take on that? It's not entirely that the algorithm does math. See, what happens is that humans still need to come up with some sort of hypothesis that two quantities are connected in some way. But then the machine is trained to learn function mapping from one quantity to the other quantity. And if the machine can do it better than chance, then that means that there is some underlying pattern right there. But the machine can also not tell the pattern explicitly, but DeepMind uses various interpretability techniques along with the results of the machine and retraining the algorithm on different subsets of features. And all of that is then given to a human mathematician to make sense of. So the humans still need to come up with a hypothesis of what could go together. And also, the humans still need to interpret the results of the algorithms to formulate really a theorem and then actually prove the theorem. The algorithm is only there to uncover new patterns and then try to give various hints on what these patterns could be. That's very interesting. So what are the results of this work? What has been achieved? So this publication has actually resulted in not one but two archive publications, both together with mathematicians in these fields. The first one is a new theorem in topology establishing a connection between the algebraic structure of knots and the geometric structure of knots. And the second one is a new hint to sort of a proof strategy for a long standing conjecture in representation theory. So does that mean that math could be solved in the near future? While these advances seem impressive, it stands to argue that this only works really for a certain subset of mathematical theorems, namely the ones where there is some sort of a pattern between two numbers that we can actually measure and the machine learning model can make sense of. Remember that mathematicians have used computers for a number of years right now to assist them. And this is simply one step more into that direction. One more class of theorems and hypotheses that are amenable to now be done by computers that help mathematicians. But it's not all of math yet. And it's arguable whether this approach will lead to all of math being solved. That is fascinating. Thank you so much, Marcus. We appreciate your input very much. Thank you very much for having me and good day. Microsoft Research Blog has a new entry called Efficiently and Effectively Scaling Up Language Model Pre-Training for Best Language Representation Model on Glue and Super Glue. The blog post is about a new model in the Microsoft Touring series called TNLRV5. This model gets state of the art on super glue and glue, which are famous NLP benchmarks. Super glue and glue themselves consist of subtasks where the model has to solve different NLP challenges. The interesting thing is that this progress hasn't been achieved by simply scaling up the models like we've seen until now, but more so by actually reducing the model size a little bit. This model in fact says that it achieves comparable effectiveness to other models with 50% fewer parameters and fewer computing cost in pre-training. It's pretty cool to see models going away from the ever bigger, ever more paradigm into the paradigm of how can we use the data and the compute that we have the most efficiently. So as you can imagine, it's not just a single idea that comes to play in here. Lots of interconnecting pieces are here, mix of scientific advances and engineering advances. They highlight a few things such as the pre-training task where a main transformer isn't necessarily fed with original text and then trying to reproduce that using language modeling, but it gets text that has been pre-corrupted by an auxiliary model. So here you can see the auxiliary transformer that gets a masked sequence and is tasked to produce a sequence out of that. So sample a sequence of text, which is then input to the main transformer. And the main transformer's job is not only to reproduce the text that has been input, but to correct for the sampling mistakes that the auxiliary model introduced. This is a bit more of an intricate version of the classic paradigm of the denoising autoencoder that we've seen during training of BERT and so on. And it seems that this task makes these models more efficient and effective with less data. They also highlight a few engineering features such as customized CUDA kernels for mixed precision training and the zero optimizer that allows models to be trained on a massively parallel architecture. A cool feature of the model is that it is not only more performant if you scale it up, but it keeps its high performance even if you scale it down, which is different from other models that only exhibit real power once you either scale them up or keep them in the low parameter regime. What's also interesting is how the model is going to be released. Microsoft says here that it's going to be released essentially as an API in Azure Cognitive Services. So that is a bit worrisome that we see more and more especially big companies going away from publishing their models instead setting up APIs, mostly paid APIs or with some sort of other attachments that lets them control their models behind a wall and lets you only access the outputs of it. Now, sure, these models are a little bit too large to run or train for most people, but still I am not sure if I'm a fan of this development. On the other hand, it is welcome that there are more and more competitors in this market of offering large scale models via APIs. That means that a single player like OpenAI doesn't have necessarily a monopoly anymore on inference on large models. If you want to know more of the details of this model, check out the blog right here, a link in the description. This is a cool website called the NeurIPS Anthology Visualization. It's based on 60 years demo from Henrik Strobold and Benjamin Hoover from MIT IBM Watson lab with data from Lee Campbell tested by Mark Aurelio Ranzato. I hope I got all the credentials right here. This is a website that interactively maps papers that have been submitted to NeurIPS and accepted, I guess, over the years since its existence. Now, not only does it map the papers and put them into a low dimensional space, it also clusters different categories together and highlights such clusters. For example, there's this cluster on papers on graphs and graph neural networks, there's a cluster on SVMs, there's a cluster on adversarial and robust learning, even one on neuroscience. Now, specifically, the color coding is the date or the year when these papers were published. And you can see a clear evolution right here. In fact, as you slide the timer here forward, you can see that the early papers were very much in the realm of neuroscience and classical neural networks slowly expanding into deep learning SVMs and then explosion all over the place into bandits and fairness and optimization and causal and reinforcement learning. While there were always papers in all of these regions, it's definitely cool to see how the conference and the entire field, by that matter, has shifted from its origins into the deep learning and general machine learning world we see today. It's also cool to see that there are still quite a few yellow dots in the neuroscience area, meaning that the true core of the conference hasn't gone missing, just kind of buried under the thousands of papers on GANs and NERF. What's also cool is that you can select a certain area, it'll show you sort of a word cloud and papers in that area, as well as a graph over time on how many papers were submitted there. And the coolest feature is that it has a text field so you can enter your abstract right here and localize your paper in the whole map of NeurIPS submissions. That's just a text field, I can enter whatever I want. I like to pick my nose. Calculating position, we're right here in the classical neural networks domain. That is very true, it is a classic problem. So let's see what our nearest neighbors here are by drawing a shape around. We have papers like a neural network approach for three dimensional object recognition. That is of course very important, like I have to recognize my nose in three dimensions. If you can see, like in two dimensions, I hit my nose every time. But in three dimensions, I completely miss it. Fast pruning is also very important because you don't want to like pick forever, you want to kind of be done very quickly. So this site is definitely, definitely worth it. If you're interested sort of in the broader landscape of machine learning research, this is an excellent site. There is a blog post going with it that details how exactly you can use the tool and what features that I haven't actually shown you so far. So definitely check that out. Our next story, Timnit Gebru launches her own research institute. The Washington Post writes in this story, Google fired its star AI researcher one year ago. Now she's launching her own institute. Now, if I understand correctly, the launching of the new institute, in fact, comes exactly one year after Gebru was fired from Google. Just for the record, I think Google would claim that Gebru left. In this article, there is a quote from Gebru saying, I've been frustrated for a long time about the incentive structures that we have in place and how none of them seem to be appropriate for the kind of work I want to do. So now she's launching her own institute. The institute is called DAIR, the Distributed AI Research Institute, and claims to be a space for independent community-rooted AI research free from big tech's pervasive influence. For now, the institute is sponsored to a tune of 3.7 million US dollars from various foundations. Gebru herself also published an opinion piece in The Guardian saying, for truly ethical AI, its research must be independent from big tech. She again recounts stories of being fired from Google and seeing firsthand the impacts that these technologies can have and the power that the big tech companies hold over it. The research institute's website states the way in which they want to perform research. They say, instead of constantly working to mitigate the harms of AI research performed by dominant groups without an analysis of potential risks and harms, we encourage a research process that analyzes its end goal and potential risks and harms from the start. The research interests of the institute are listed here, developing AI for low resource settings, language technology serving marginalized communities, coordinated social media activity, data-related work, and robustness testing and documentation. In one of the articles, I also saw a word about low resource languages, and as a speaker of Swiss German, I fully approve. We don't even have a written form. Now, honestly, I find this to be a pretty good solution instead of people that have problems with how big tech conducts research, just sort of shooting against big tech and complaining about it. Now they get the opportunity to actually make research as they see fit. And if it turns out well, then it's, I guess, all the better. Now, it is a hard task to invent new things, to actually create new things while also having all these things in mind. That is a pretty difficult problem. That's why we historically had people sort of pushing technology ahead and then other people cleaning up after them and sort of making the already existing technology better, more accessible, more fair, and so on. This research institute's goal seemed to do all of these things jointly. And yeah, I look forward to what comes out of it. And being funded through foundations, of course, relieves some of the stress of big tech, which always has to essentially make more and more profit. The question is, of course, a little bit what happens when this money runs out? What happens if the sponsors themselves come and impose some restrictions on the research institute? What if they want their interests to be represented in the research? I guess even with foundation money, it doesn't come without any strings attached. It's not as easy as it seems, but it's different. And I think that's good. Amazon announces SageMaker Canvas, which is sort of a no-code machine learning platform on SageMaker. As you can see, they have a few screenshots of the user interface with interesting animated characters. You can import your data, look at it, analyze it, and then you can train some machine learning models. But here we go. We're doing some analytics on it. We train some classifier. Look, we got a 99.9% estimated accuracy. Oh, wow. That is amazing. We can then analyze these models that we've trained on various other things and ultimately ship them out. And all of this without writing a single line of code. So no code seems to be a coming business, especially, I guess, targeted towards people who might know how to do a little bit of pandas, but might not be as versed in actual machine learning. And given that training simple models has become quite an easy task to do now, it makes sense to integrate this into a nice GUI and make it accessible to a lot more people. All right. Quick series of helpful things. I guess this section was termed helpful libraries at one point. We'll have to rename it. You just like help, help, like double help, help, help, helpful things and more. MacBIRTH is a series of BERT models pre-trained on historical textual material. The date ranges from 1450 to 1950. If you want some ye older language, you can find it in the hogging face repository. NVIDIA announces TensorRT 8.2, which is a library that makes machine learning models run faster on NVIDIA hardware. And the cool thing about this release is the direct integrations with TensorFlow and PyTorch. So rather than going through an arduous process of converting your model from your format to their format, you can get a lot of the speed ups already by a single line of code. For example, they say integration for PyTorch delivers up to 6x performance versus in framework inference on GPUs with just one line of code. And the same goes for TensorFlow. Opacus released version 1.0. It is a library to train PyTorch models with differential privacy. Now, what I love is how easy all these libraries make it look like. So you got your standard neural net and optimizer and data loader. Then you load up a privacy engine. And all you do is you say, make private. And then they say, now it's business as usual. Seems pretty easy. Whether or not that works out in practice, I don't know. But if you're looking into differential privacy, this seems like a very good point to start. This is clip guided collage, which allows you to give clip a bunch of these individual elements, in this case, fruit, and then let clip generate a collage from them. I guess this is supposed to be a smiley face at the end, but there are lots of cool examples all over. I mean, it just looks really funky. There is a cool app if you want to play around with it. And shout out to Nao Tokui for creating it. Thomas Simonini writes, we just published Snowball Fight, the first hugging face deep reinforcement learning environment. So this is based on the Unity engine. It's an RL environment, but it is in 3D and you can play it. So I'll be Clem the Duck. And this is against an agent that's been pre-trained with, I believe, proximal policy optimization. Now, I have tried this before, but it's not that easy. You get sort of this ouch, ouch, haha. Oh crap, I died. Um, if you want to try it out, you can try it out on the hugging face hub directly or you train an RL agent for it. Archive Sanity Lite is a new iteration of Archive Sanity. It's by Andrej Karpati and you have the ability to self-host this system or there is a version running online. Archive Sanity famously is a system where you can enter your personal preferences, tags, favorite papers, and so on. And it will suggest you out of new archive publications, which ones you might like most. This is definitely a good way to make sense out of the flood of archive papers that come in every single day. If you liked my video about backpropagating through discrete black box algorithms, you might also like this related paper, Learning with Algorithmic Supervision via Continuous Relaxations. This is a bit of a different approach, but it also allows you to work with algorithms within the layers of neural networks. The video is by Felix Peterson and I'll link to it in the description. Koila is a library that prevents CUDA out of memory errors with one single line of code. So what you do is you wrap your mini-batches inside of this library and the library will decide itself how much to lazily compute through the network. So as you can see, all you have to do is you wrap your input and label tensors in this lazy function and off you go. If you liked my video about Efficient Zero, the code for it has now been open source. Check it out. Shout out to CorneliusMD that won the 3090 of our giveaway. Congratulations, Cornelius, and I'm sorry to everyone else. I hope we can make some giveaways in the future as well. Looks quite pretty, doesn't it? And lastly, there is a NURIPS blog post called A Retrospective on the NURIPS 2021 Ethics Review Process. NURIPS has ramped up its ethics review, including much more papers in the review process, recruiting much more reviewers, and this blog post is a reflection on that process. From the statistics, you can see that a couple of hundred papers, like two or three hundred papers, were ultimately flagged for ethic review. Precisely it was 265 papers out of 9,122 submissions. One interesting fact is that whenever two ethics reviewers were assigned per paper, and I think that was the default, they often didn't necessarily agree whether or not there were ethical issues with the paper. To give some of the examples here of the identified issues, lack of sufficient reflection around topics that involve thorny ethical considerations, the use of deprecated data sets that had been explicitly removed by their authors, lack of transparency on model or data details, among other things, a lack of communications on the details of annotator work conditions, but also things like violating copyright restrictions and the lack of sending the project through an institutional review board in situations clearly involving human subjects, and lastly, uncritically emphasizing explicitly harmful applications such as police profiling. They say in some cases the concerns raised were so critical that the acceptance of the paper was made conditional on the authors implementing the suggested mitigations. All such cases were discussed by the program chairs and ethics review chairs, and the ethics reviewers were consulted in determining conditions for acceptance. Of eight papers conditionally accepted for ethical reasons, all were eventually accepted. They also say in a single case, the program chairs and ethics review chairs jointly determined that the required mitigations would be so challenging to execute that they were beyond the scope of what the authors could realistically accomplish within the timeframe for the camera ready. In this case, the program chairs made the call to reject the paper on ethical grounds. So ultimately, one paper was rejected and a bunch of papers were forced to put something in that wasn't originally in. But now what I find interesting here is that again, not even the ethics reviewers necessarily agree among themselves what is an ethical issue and what is not, which is a consequence of there being much more ethics reviewers this year, I believe than last year, and therefore, I guess also a more diverse set of opinions. Now, this is both a good thing, since I believe more diverse opinions make the field richer, but also a little bit of a bad thing as we now carry over the absolutely noisy random review process from the regular review over to the ethics review where papers are hit by yet another completely random or semi random process. It's fair to say that the same issues appear here when you try to scale up these ethics reviews as when you try to scale up the normal reviews. My other concern is that while some of the ethics violations are probably less controversial, there are also clearly political ethics violations discussed right here. And I'm not entirely sure if that is a direction that the field wants to go to take very strong positions on things rather than remaining neutral. I guess it's not a solved issue and the degree to which this is important has to be figured out by the community. We'll see what happens in the following years. All right, that was already it for ML News. Thank you so much for being here. Check out Weights and Biases, get enough sleep, and I'll see you next time. Bye bye.
 DeepMind tackles fundamental mathematics, Microsoft trains its most efficient and effective language model yet,
 and Timniggebru launches her own research institute.
 Welcome to ML News!
 Look at this, look at what I got as a Christmas present.
 It is a swag package from Weights and Biases.
 So, so if you look, there's lots of like yellow fuzzy fuzzy stuff to package, but mainly these are socks,
 Weights and Biases themed socks.
 Look at that. It's Weights and Biases socks.
 They have like little B's and little ones.
 Oh, I get it.
 Now you can see me here actually on camera realizing the following.
 See Weights and Biases URL is 1db.com.
 It's W and B.
 Now I have not realized this before, but the 1d and the B obviously stand for this URL.
 Now you can see me realize this right here on camera.
 Watch.
 It's 1db, like a 1d and a B.
 I just got this right, like literally I did not get this until right now.
 A 1d and a B.
 And then most importantly, this thing right here, which is a...
 Mug.
 Excellent.
 And this is really cool. Look at that.
 Like it's a colorless logo. It's kind of imprinted in metal.
 This is very cool cup.
 One sec.
 All right. I filled this up with tea.
 It is actually still steaming.
 It's completely hot on the inside, completely cool on the outside.
 Excellent.
 Thank you very much Weights and Biases for this awesome Christmas gift.
 Coincidentally, this video is sponsored by Weights and Biases.
 If you don't know Weights and Biases yet, please go check them out.
 Weights and Biases is the tool for your machine learning needs.
 It can do experiment tracking.
 One line of code tracks your experiments to the cloud.
 Nicely viewable.
 For every experiment, you can save all the output, all the logs, all the graphs.
 You can compare experiments.
 Weights and Biases can track your data sets and your models
 and save them as artifacts in the cloud.
 You'll know exactly how to reproduce every single thing there is.
 They have a really neat feature called tables where you can analyze your data,
 filter it, and really go into the depth of where your models still need improvement.
 This is not only useful during experimentation.
 It's actually useful all the way to deployment and monitoring
 after you've deployed your model.
 And then lastly, you can also pull all of this into reports,
 which is an interactive document that you can send to your boss,
 your team members, your clients even, and show them interactively
 how their stuff is doing.
 Reports are living documents with interactive plots and tables
 and all of the other features.
 So if you still do ML tooling by hand, give Weights and Biases a try.
 It's completely free for personal use and for academic use.
 They have solutions on cloud and on premise.
 There's no excuse not to check them out.
 Again, thank you so much, Weights and Biases, for sponsoring this video,
 for the awesome gift package.
 As you see, I am very bribable.
 And let's get into the video.
 DeepMind has a new blog post called Exploring the Beauty of Pure Mathematics
 in Novel Ways.
 And this blog post goes along with a paper in the journal Nature
 called Advancing Mathematics by Guiding Human Intuition with AI.
 This is a joint effort by DeepMind scholars and people in the actual
 mathematical fields to use AI to make new mathematical discoveries.
 Now, by new mathematical discoveries, I don't mean like the last digit of pi
 or something like this.
 These are actual fundamental theorems in fields like topology.
 Now, because I'm pretty bad at fundamental math, right now I'm actually
 going to speak to an outside correspondent who gives us the details on this story.
 I'm speaking live to Marcus Bedding.
 Marcus, it's very nice to have you on the show.
 Hi, Onik. Thanks for having me. Nice to be on the show.
 In fact, I'm standing in front of the building where math was once performed,
 apparently.
 So, Marcus, tell us, has DeepMind solved math?
 Is AI doing math now?
 Are mathematicians going to be obsolete?
 What's your take on that?
 It's not entirely that the algorithm does math.
 See, what happens is that humans still need to come up with some sort of
 hypothesis that two quantities are connected in some way.
 But then the machine is trained to learn function mapping from one quantity
 to the other quantity.
 And if the machine can do it better than chance, then that means that there is
 some underlying pattern right there.
 But the machine can also not tell the pattern explicitly, but DeepMind uses
 various interpretability techniques along with the results of the machine and
 retraining the algorithm on different subsets of features.
 And all of that is then given to a human mathematician to make sense of.
 So the humans still need to come up with a hypothesis of what could go together.
 And also, the humans still need to interpret the results of the algorithms
 to formulate really a theorem and then actually prove the theorem.
 The algorithm is only there to uncover new patterns and then try to give various
 hints on what these patterns could be.
 That's very interesting.
 So what are the results of this work?
 What has been achieved?
 So this publication has actually resulted in not one but two archive publications,
 both together with mathematicians in these fields.
 The first one is a new theorem in topology establishing a connection between
 the algebraic structure of knots and the geometric structure of knots.
 And the second one is a new hint to sort of a proof strategy for
 a long standing conjecture in representation theory.
 So does that mean that math could be solved in the near future?
 While these advances seem impressive, it stands to argue that this only works
 really for a certain subset of mathematical theorems, namely the ones
 where there is some sort of a pattern between two numbers that we can actually
 measure and the machine learning model can make sense of.
 Remember that mathematicians have used computers for
 a number of years right now to assist them.
 And this is simply one step more into that direction.
 One more class of theorems and
 hypotheses that are amenable to now be done by computers that help mathematicians.
 But it's not all of math yet.
 And it's arguable whether this approach will lead to all of math being solved.
 That is fascinating.
 Thank you so much, Marcus.
 We appreciate your input very much.
 Thank you very much for having me and good day.
 Microsoft Research Blog has a new entry called Efficiently and
 Effectively Scaling Up Language Model Pre-Training for
 Best Language Representation Model on Glue and Super Glue.
 The blog post is about a new model in the Microsoft Touring series called TNLRV5.
 This model gets state of the art on super glue and glue,
 which are famous NLP benchmarks.
 Super glue and glue themselves consist of subtasks where the model has to solve
 different NLP challenges.
 The interesting thing is that this progress hasn't been achieved by simply
 scaling up the models like we've seen until now, but more so by actually
 reducing the model size a little bit.
 This model in fact says that it achieves comparable effectiveness to other models
 with 50% fewer parameters and fewer computing cost in pre-training.
 It's pretty cool to see models going away from the ever bigger,
 ever more paradigm into the paradigm of how can we use the data and the compute
 that we have the most efficiently.
 So as you can imagine, it's not just a single idea that comes to play in here.
 Lots of interconnecting pieces are here, mix of scientific advances and
 engineering advances.
 They highlight a few things such as the pre-training task where a main
 transformer isn't necessarily fed with original text and then trying to
 reproduce that using language modeling, but it gets text that has been
 pre-corrupted by an auxiliary model.
 So here you can see the auxiliary transformer that gets a masked sequence
 and is tasked to produce a sequence out of that.
 So sample a sequence of text, which is then input to the main transformer.
 And the main transformer's job is not only to reproduce the text that has been
 input, but to correct for the sampling mistakes that the auxiliary model
 introduced.
 This is a bit more of an intricate version of the classic paradigm of the
 denoising autoencoder that we've seen during training of BERT and so on.
 And it seems that this task makes these models more efficient and effective with
 less data.
 They also highlight a few engineering features such as customized CUDA
 kernels for mixed precision training and the zero optimizer that allows models
 to be trained on a massively parallel architecture.
 A cool feature of the model is that it is not only more performant if you scale
 it up, but it keeps its high performance even if you scale it down,
 which is different from other models that only exhibit real power once you
 either scale them up or keep them in the low parameter regime.
 What's also interesting is how the model is going to be released.
 Microsoft says here that it's going to be released essentially as an API in
 Azure Cognitive Services.
 So that is a bit worrisome that we see more and more especially big companies
 going away from publishing their models instead setting up APIs,
 mostly paid APIs or with some sort of other attachments that lets them control
 their models behind a wall and lets you only access the outputs of it.
 Now, sure, these models are a little bit too large to run or train for most
 people, but still I am not sure if I'm a fan of this development.
 On the other hand, it is welcome that there are more and more competitors in
 this market of offering large scale models via APIs.
 That means that a single player like OpenAI doesn't have necessarily a
 monopoly anymore on inference on large models.
 If you want to know more of the details of this model, check out the blog right
 here, a link in the description.
 This is a cool website called the NeurIPS Anthology Visualization.
 It's based on 60 years demo from Henrik Strobold and Benjamin Hoover from MIT
 IBM Watson lab with data from Lee Campbell tested by Mark Aurelio Ranzato.
 I hope I got all the credentials right here.
 This is a website that interactively maps papers that have been submitted to
 NeurIPS and accepted, I guess, over the years since its existence.
 Now, not only does it map the papers and put them into a low dimensional space,
 it also clusters different categories together and highlights such clusters.
 For example, there's this cluster on papers on graphs and graph neural
 networks, there's a cluster on SVMs, there's a cluster on adversarial and
 robust learning, even one on neuroscience.
 Now, specifically, the color coding is the date or the year when these papers
 were published.
 And you can see a clear evolution right here.
 In fact, as you slide the timer here forward, you can see that the early
 papers were very much in the realm of neuroscience and classical neural
 networks slowly expanding into deep learning SVMs and then explosion all over
 the place into bandits and fairness and optimization and causal and
 reinforcement learning.
 While there were always papers in all of these regions, it's definitely cool to
 see how the conference and the entire field, by that matter, has shifted from
 its origins into the deep learning and general machine learning world we see
 today.
 It's also cool to see that there are still quite a few yellow dots in the
 neuroscience area, meaning that the true core of the conference hasn't gone
 missing, just kind of buried under the thousands of papers on GANs and NERF.
 What's also cool is that you can select a certain area, it'll show you sort of a
 word cloud and papers in that area, as well as a graph over time on how many
 papers were submitted there.
 And the coolest feature is that it has a text field so you can enter your
 abstract right here and localize your paper in the whole map of NeurIPS
 submissions.
 That's just a text field, I can enter whatever I want.
 I like to pick my nose.
 Calculating position, we're right here in the classical neural networks domain.
 That is very true, it is a classic problem.
 So let's see what our nearest neighbors here are by drawing a shape around.
 We have papers like a neural network approach for three dimensional object
 recognition.
 That is of course very important, like I have to recognize my nose in three
 dimensions.
 If you can see, like in two dimensions, I hit my nose every time.
 But in three dimensions, I completely miss it.
 Fast pruning is also very important because you don't want to like pick
 forever, you want to kind of be done very quickly.
 So this site is definitely, definitely worth it.
 If you're interested sort of in the broader landscape of machine learning
 research, this is an excellent site.
 There is a blog post going with it that details how exactly you can use the tool
 and what features that I haven't actually shown you so far.
 So definitely check that out.
 Our next story, Timnit Gebru launches her own research institute.
 The Washington Post writes in this story, Google fired its star AI researcher
 one year ago.
 Now she's launching her own institute.
 Now, if I understand correctly, the launching of the new institute, in fact,
 comes exactly one year after Gebru was fired from Google.
 Just for the record, I think Google would claim that Gebru left.
 In this article, there is a quote from Gebru saying, I've been frustrated for a
 long time about the incentive structures that we have in place and how none of
 them seem to be appropriate for the kind of work I want to do.
 So now she's launching her own institute.
 The institute is called DAIR, the Distributed AI Research Institute, and
 claims to be a space for independent community-rooted AI research free from
 big tech's pervasive influence.
 For now, the institute is sponsored to a tune of 3.7 million US dollars from
 various foundations.
 Gebru herself also published an opinion piece in The Guardian saying, for truly
 ethical AI, its research must be independent from big tech.
 She again recounts stories of being fired from Google and seeing firsthand
 the impacts that these technologies can have and the power that the big tech
 companies hold over it.
 The research institute's website states the way in which they want to perform
 research.
 They say, instead of constantly working to mitigate the harms of AI research
 performed by dominant groups without an analysis of potential risks and harms, we
 encourage a research process that analyzes its end goal and potential risks
 and harms from the start.
 The research interests of the institute are listed here, developing AI for low
 resource settings, language technology serving marginalized communities,
 coordinated social media activity, data-related work, and robustness testing
 and documentation.
 In one of the articles, I also saw a word about low resource languages, and as a
 speaker of Swiss German, I fully approve.
 We don't even have a written form.
 Now, honestly, I find this to be a pretty good solution instead of people that have
 problems with how big tech conducts research, just sort of shooting against big
 tech and complaining about it.
 Now they get the opportunity to actually make research as they see fit.
 And if it turns out well, then it's, I guess, all the better.
 Now, it is a hard task to invent new things, to actually create new things
 while also having all these things in mind.
 That is a pretty difficult problem.
 That's why we historically had people sort of pushing technology ahead and then
 other people cleaning up after them and sort of making the already existing
 technology better, more accessible, more fair, and so on.
 This research institute's goal seemed to do all of these things jointly.
 And yeah, I look forward to what comes out of it.
 And being funded through foundations, of course, relieves some of the stress of
 big tech, which always has to essentially make more and more profit.
 The question is, of course, a little bit what happens when this money runs out?
 What happens if the sponsors themselves come and impose some restrictions on the
 research institute?
 What if they want their interests to be represented in the research?
 I guess even with foundation money, it doesn't come without any strings attached.
 It's not as easy as it seems, but it's different.
 And I think that's good.
 Amazon announces SageMaker Canvas, which is sort of a no-code machine learning
 platform on SageMaker.
 As you can see, they have a few screenshots of the user interface with
 interesting animated characters.
 You can import your data, look at it, analyze it, and then you can train some
 machine learning models.
 But here we go.
 We're doing some analytics on it.
 We train some classifier.
 Look, we got a 99.9% estimated accuracy.
 Oh, wow.
 That is amazing.
 We can then analyze these models that we've trained on various other things and
 ultimately ship them out.
 And all of this without writing a single line of code.
 So no code seems to be a coming business, especially, I guess, targeted towards
 people who might know how to do a little bit of pandas, but might not be as versed
 in actual machine learning.
 And given that training simple models has become quite an easy task to do now, it
 makes sense to integrate this into a nice GUI and make it accessible to a lot more
 people.
 All right.
 Quick series of helpful things.
 I guess this section was termed helpful libraries at one point.
 We'll have to rename it.
 You just like help, help, like double help, help, help, helpful things and more.
 MacBIRTH is a series of BERT models pre-trained on historical textual material.
 The date ranges from 1450 to 1950.
 If you want some ye older language, you can find it in the hogging face
 repository.
 NVIDIA announces TensorRT 8.2, which is a library that makes machine learning
 models run faster on NVIDIA hardware.
 And the cool thing about this release is the direct integrations with TensorFlow
 and PyTorch.
 So rather than going through an arduous process of converting your model from
 your format to their format, you can get a lot of the speed ups already by a
 single line of code.
 For example, they say integration for PyTorch delivers up to 6x performance
 versus in framework inference on GPUs with just one line of code.
 And the same goes for TensorFlow.
 Opacus released version 1.0.
 It is a library to train PyTorch models with differential privacy.
 Now, what I love is how easy all these libraries make it look like.
 So you got your standard neural net and optimizer and data loader.
 Then you load up a privacy engine.
 And all you do is you say, make private.
 And then they say, now it's business as usual.
 Seems pretty easy.
 Whether or not that works out in practice, I don't know.
 But if you're looking into differential privacy, this seems like a very good
 point to start.
 This is clip guided collage, which allows you to give clip a bunch of these
 individual elements, in this case, fruit, and then let clip generate a collage
 from them.
 I guess this is supposed to be a smiley face at the end, but there are lots of
 cool examples all over.
 I mean, it just looks really funky.
 There is a cool app if you want to play around with it.
 And shout out to Nao Tokui for creating it.
 Thomas Simonini writes, we just published Snowball Fight, the first hugging
 face deep reinforcement learning environment.
 So this is based on the Unity engine.
 It's an RL environment, but it is in 3D and you can play it.
 So I'll be Clem the Duck.
 And this is against an agent that's been pre-trained with, I believe, proximal
 policy optimization.
 Now, I have tried this before, but it's not that easy.
 You get sort of this ouch, ouch, haha.
 Oh crap, I died.
 Um, if you want to try it out, you can try it out on the hugging face hub
 directly or you train an RL agent for it.
 Archive Sanity Lite is a new iteration of Archive Sanity.
 It's by Andrej Karpati and you have the ability to self-host this system or there is a version
 running online.
 Archive Sanity famously is a system where you can enter your personal preferences,
 tags, favorite papers, and so on.
 And it will suggest you out of new archive publications, which ones you might like most.
 This is definitely a good way to make sense out of the flood of archive papers that come
 in every single day.
 If you liked my video about backpropagating through discrete black box algorithms, you
 might also like this related paper, Learning with Algorithmic Supervision via Continuous
 Relaxations.
 This is a bit of a different approach, but it also allows you to work with algorithms
 within the layers of neural networks.
 The video is by Felix Peterson and I'll link to it in the description.
 Koila is a library that prevents CUDA out of memory errors with one single line of code.
 So what you do is you wrap your mini-batches inside of this library and the library will
 decide itself how much to lazily compute through the network.
 So as you can see, all you have to do is you wrap your input and label tensors in this
 lazy function and off you go.
 If you liked my video about Efficient Zero, the code for it has now been open source.
 Check it out.
 Shout out to CorneliusMD that won the 3090 of our giveaway.
 Congratulations, Cornelius, and I'm sorry to everyone else.
 I hope we can make some giveaways in the future as well.
 Looks quite pretty, doesn't it?
 And lastly, there is a NURIPS blog post called A Retrospective on the NURIPS 2021 Ethics
 Review Process.
 NURIPS has ramped up its ethics review, including much more papers in the review process, recruiting
 much more reviewers, and this blog post is a reflection on that process.
 From the statistics, you can see that a couple of hundred papers, like two or three hundred
 papers, were ultimately flagged for ethic review.
 Precisely it was 265 papers out of 9,122 submissions.
 One interesting fact is that whenever two ethics reviewers were assigned per paper,
 and I think that was the default, they often didn't necessarily agree whether or not there
 were ethical issues with the paper.
 To give some of the examples here of the identified issues, lack of sufficient reflection around
 topics that involve thorny ethical considerations, the use of deprecated data sets that had been
 explicitly removed by their authors, lack of transparency on model or data details,
 among other things, a lack of communications on the details of annotator work conditions,
 but also things like violating copyright restrictions and the lack of sending the project through
 an institutional review board in situations clearly involving human subjects, and lastly,
 uncritically emphasizing explicitly harmful applications such as police profiling.
 They say in some cases the concerns raised were so critical that the acceptance of the
 paper was made conditional on the authors implementing the suggested mitigations.
 All such cases were discussed by the program chairs and ethics review chairs, and the ethics
 reviewers were consulted in determining conditions for acceptance.
 Of eight papers conditionally accepted for ethical reasons, all were eventually accepted.
 They also say in a single case, the program chairs and ethics review chairs jointly determined
 that the required mitigations would be so challenging to execute that they were beyond
 the scope of what the authors could realistically accomplish within the timeframe for the camera
 ready.
 In this case, the program chairs made the call to reject the paper on ethical grounds.
 So ultimately, one paper was rejected and a bunch of papers were forced to put something
 in that wasn't originally in.
 But now what I find interesting here is that again, not even the ethics reviewers necessarily
 agree among themselves what is an ethical issue and what is not, which is a consequence
 of there being much more ethics reviewers this year, I believe than last year, and therefore,
 I guess also a more diverse set of opinions.
 Now, this is both a good thing, since I believe more diverse opinions make the field richer,
 but also a little bit of a bad thing as we now carry over the absolutely noisy random
 review process from the regular review over to the ethics review where papers are hit
 by yet another completely random or semi random process.
 It's fair to say that the same issues appear here when you try to scale up these ethics
 reviews as when you try to scale up the normal reviews.
 My other concern is that while some of the ethics violations are probably less controversial,
 there are also clearly political ethics violations discussed right here.
 And I'm not entirely sure if that is a direction that the field wants to go to take very strong
 positions on things rather than remaining neutral.
 I guess it's not a solved issue and the degree to which this is important has to be figured
 out by the community.
 We'll see what happens in the following years.
 All right, that was already it for ML News.
 Thank you so much for being here.
 Check out Weights and Biases, get enough sleep, and I'll see you next time.
 Bye bye.