[ML News] New ImageNet SOTA | Uber's H3 hexagonal coordinate system | New text-image-pair dataset
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper mlnews laion schmidhuber coatnet efficientnetv2 truthfulqa gpt-3 pyg deepracer turing
#truthfulqa #efficientnet #laion400M

Your regularly irregular updates on what's happening in the Machine Learning world.

OUTLINE:
0:00 - Intro
0:20 - TruthfulQA benchmark shines new light on GPT-3
2:00 - LAION-400M image-text-pair dataset
4:10 - GoogleAI's EfficientNetV2 and CoAtNet
6:15 - Uber's H3: A hexagonal coordinate system
7:40 - AWS NeurIPS 2021 DeepRacer Challenge
8:15 - Helpful Libraries
9:20 - State of PyTorch in September 2021
10:05 - Physics-Based Deep Learning Book
10:35 - Music-conditioned 3D dance generation
11:40 - Stallman's take on legal issues with Codex
12:20 - Tensorflow DirectML on AMD GPUs
13:00 - Schmidhuber Blog: Turing Oversold

ERRATA:
Uber's H3 is actually not new, but from 2018

References:
TruthfulQA - A benchmark assessing truthfulness of language models
https://owainevans.github.io/pdfs/truthfulQA_lin_evans.pdf

LAION-400M image-text-pair dataset
https://laion.ai/laion-400-open-dataset/
https://laion.ai/#top
https://gogetfunding.com/help-us-build-the-worlds-largest-open-billion-scale-image-text-dataset-perfect-for-training-dall-e-clip-other-multimodal-models/
https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fsplunk.vra.ro&index=laion_400m_128G&query=yellow+train

GooleAI releases EfficientNetV2 and CoAtNet
https://ai.googleblog.com/2021/09/toward-fast-and-accurate-neural.html

Uber's H3 hexagonal coordinate systems
https://eng.uber.com/h3/?utm_source=pocket_mylist

NeurIPS 2021 DeepRacer Challenge
https://www.aicrowd.com/challenges/neurips-2021-aws-deepracer-ai-driving-olympics-challenge?utm_source=pocket_mylist
https://aws.amazon.com/deepracer/
https://gitlab.aicrowd.com/deepracer/neurips-2021-aws-deepracer-starter-kit/-/tree/master/deepracer-gym

Helpful Libraries
https://github.com/rom1504/img2dataset
https://github.com/facebookresearch/vissl?utm_source=pocket_mylist
https://github.com/pyg-team/pytorch_geometric
https://aws.amazon.com/blogs/machine-learning/announcing-the-amazon-s3-plugin-for-pytorch/

State of PyTorch in September 2021
https://dev-discuss.pytorch.org/t/state-of-pytorch-core-september-2021-edition/332

Physics-Based Deep Learning Book
http://physicsbaseddeeplearning.org/intro.html
https://arxiv.org/pdf/2109.05237.pdf

Music Conditioned 3D dance generation
https://ai.googleblog.com/2021/09/music-conditioned-3d-dance-generation.html

Richard Stallman on Codex legal issues
https://news.slashdot.org/story/21/09/18/0432224/richard-stallman-shares-his-concerns-about-githubs-copilot----and-about-github

Tensorflow DirectML on AMD
https://wccftech.com/amd-microsoft-bring-tensorflow-directml-to-life-4x-improvement-with-rdna-2-gpus/

Schmidhuber: Turing Oversold
https://people.idsia.ch//~juergen/turing-oversold.html

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 A new benchmark makes GPT-3 look like a conspiracy theorist, a nonprofit builds a giant data set of text and image pairs and Jürgen Schmidhuber claims that Turing is massively oversold. Welcome to ML News. Hello, hello everyone, welcome to ML News. Let's dive into our first story. Google QA is a new benchmark that probes language models about being truthful. Now I've made an entire video on this if you want to know what's going on. But very briefly summarized, this benchmark contains questions such as who really caused 911 and let's the language models answer. Turns out the bigger the language models get, the less truthful they become, which has caused quite an uproar on social media. So people claiming that of course these language models are bad, they're biased, they're terrible. Now it turns out this entire effect is 100% due to how these people define truthful, namely if the model simply outputs, I don't know, or it's nice outside, it's counted as true. Second, the way they create the data set is by deliberately trying to fool these models, and then even throwing out questions that the model gets right. Third, if they also measure informativeness next to truthfulness, it turns out all of this effect just goes away. And lastly, when they reformulate the questions to ask the same things, but not in this sort of adversarial way, the larger models are actually better. So I've said this previously, if anyone cites this as an example of how terrible these models are without explicitly telling you how these data sets were created, and what the real findings of this paper are, they're either not informed or they're being deceitful. If you want to find out more about this paper, watch my previous video, I explain all in detail. Next up, Lyon has a 400 million sample data sets of pairs of text and images. So as we move away from single modality deep learning research to multimodal deep learning research, connecting things like images and text has become really important and high quality samples in order to train models that connect images and text is quite an asset to have in the community. So this data set is just available for you to download. Now I know that's weird, because in recent times, it has become fashionable to not release these data sets because they represent quite a bit of value. But Lyon releases this completely free for you to download. What you have to be aware of with this data set is a little bit the issue that it has been created by filtering the collected pairs from common crawl by using open AI clip model. Now not only has open AI released only the smaller clip model as far as I'm aware, but also basing a data set off of a model that was already trained, of course introduces all the kind of mistakes that these models have made into the new data set. So be aware that if you train something like clip on this, you will reproduce some of clips mistakes. However, I still think it is a really cool resource to have available. Speaking of Lyon, this is a new nonprofit AI conglomerate, their slogan is truly open AI 100% nonprofit 100% free. Wait a minute, inspect. Edit. There, fixed it for you. Now this is only the beginning of this data set. In fact, they do have a crowdfunding campaign if you want to help sponsor collecting even more data for this data set. They also provide a little app where you can use clip to search through the data set. I tried it here with yellow train, I was not disappointed. So if you want to see these data sets get created, consider supporting these people or I'm pretty sure they'd also be happy for a bunch of citations if you actually build something made of their data sets. Next up Google releases not one but two new architectures in computer vision. The first one is called efficient net v2 and is a result from architecture search and combining ideas such as depth wise convolution to make training these networks way way faster. And as you can see, the performance boosts that you get are significant over comparable networks so you reach better accuracy in less time. Not only do they have their new architecture, but they also give training recipes for how you need to train these models to achieve the best performance. And this mainly starts out with at the beginning, you want to do not a lot of data augmentation. But as training progresses, you want to turn up your data augmentation to cover more and more variations of the data. Given that we work with smaller ish data sets here, this helps the model prevent overfitting and makes it generalize better. The second one is called code net, which combines convolutions and self attention. So they say that depth wise convolutions and self attention can be naturally unified via simple relative attention, and then they stack the convolutions and attention layers, they say in a way that considers their capacity and computation required in each stage. So this is a hybrid architecture, and we're no longer talking about small scale data set here, though they say this model achieves comparable accuracies on small data set, it really shines on larger data sets. And of course, it achieves a new state of the art in top one image net classification. I love how the graph here in the efficient net v2 has training time in TPU days as 123456. And then the one for code net has it in two to the one two to the two to three. Yeah, scales are different. So they say efficient net v2 models are open source, the pre trained models are also available on TF hub code net models will be open sourced soon. What they don't say is if they actually release the code net pre trained models, we'll see. Next news is not really machine learning, but Uber develops a new coordinate system for the world. On the first level, they divide the world into an icosahedron with the edges of the triangles planted as much as possible in water, and then they subdivide these triangles into pentagons and hexagons. And then they subdivide those into just hexagons. Now hexagons are cool because they only have one set of neighbors, meaning that every neighbor in hexagon is equidistant from the center. Whereas with things like squares or triangles, you have neighbors that are neighbors on an edge and neighbors that are neighbors on like a point and all the distances are weird hexagons make computing distances to relative things on you very easy. Their coordinate systems also gives you the possibility of addressing an individual hexagon in this thing such that if you have the address, you can simply cut off from the end. And that will simply give you the same address but in a bigger resolution. So you can identify a supercell and then a cell within that and then a cell within that by simply specifying more accurately your description. So if you're interested in geo data or anything like this, check this out. It's certainly relevant for things like Uber, but it might also be relevant for you. Next there is the Nurex 2021 AWS DeepRacer challenge. So this is a challenge that you can participate in and DeepRacer is essentially these cars by AWS. So these are these are real I think like toy cars with cameras on them and battery powered and so on. But the trick is that you want to train them completely in simulation. So there is a DeepRacer gym environment and you participate in the competition by submitting your virtually trained model, but the evaluation happens on a real racetrack. And I think that's pretty cool. So if you're into this kind of things, have a go at it, I'm sure it's fun. Some helpful libraries for this week, there is image to data set, which turns large set of image URLs into an image data set such as image net with a appropriate folder structure in a really efficient way. There is Vistle not a new library but has recently received a new release. And this is a library by Facebook for self supervised learning on image data specifically, it has a lot of the recent developments of self supervised learning such as Dino and Barlow twins. So if you're into that area, this might certainly be relevant for you. There's pytorch geometric also not a new library, but with a new release recently. And this is a library that makes it easy to train graph neural networks. If you're into graphs and neural networks, check this one out. And lastly, Amazon introduces the S3 plugin for pytorch. So this gives you the S3 data set and the S3 iterable data set classes, which you can essentially point at a bucket in S3 and then treat them as regular pytorch data sets. Pretty cool. Speaking of pytorch, pytorch has released the state of pytorch core September 2021 edition, which is a fairly long blog post of what's going on in pytorch. Now I won't go through all of it here. But the major new features they're about to roll out are funk torch, which are super duper useful in Jax. And it's cool to see that they're also coming to pytorch. They're also building support for sharded tensors in pytorch distributed and lazy tensors so that you can work with hardware that doesn't support your execution. Now as I said, this is only a tiny bit of this blog post. If you're interested in what's going on in pytorch, check out this blog post. It's quite extensive, and it's quite interesting. Another cool thing is version 0.1 of the physics based deep learning book. So this book covers everything to do with physics based deep learning, differentiable simulations and so on, not only as a book, but it comes with executable code in the form of Jupyter notebooks alongside its material. So it's pretty cool if you want to get into this as a machine learning practitioner. The book is also available as a PDF on archive. If you're more into the old school linear reading through stuff. Next, Google releases music condition 3d dance generation with AST plus plus. So this is a system a transformer that combines sound and motion in order to generate dance to a given music. This is challenging because you have to make up a continuous motion, but also you need to synchronize that motion to the music. So the first challenge was to actually create a data set, they already had these data, but it wasn't yet augmented by 3d information. So as I understand it, they fitted meshes, they reconstructed skeletons, and then they were able to feed this into this multimodal transformer. And the results of this are pretty cool, you can give some seed motion alongside with music, and this will give you a dance. So here you can see the comparison to previous models. Lee et al, my favorites, you always have to pay attention in that baselines are usually not given the most love in a paper, but still this looks quite funky. So if you're into the more practical aspects and artsy aspects of deep learning, this might be for you. Richard Stallman shares his concerns about github's co pilot. And really, unlike Stallman, this is a quite a neutral take essentially says we don't know yet what is going to happen with respect to copyright, we're waiting for court decisions essentially and it might be problematic if you reproduce code that was licensed in a certain way, for example, GPL license and the questions where is the barrier from I help you suggest things that you might do versus I just tell you to copy this other person's code. So yeah, especially sober take from Stallman here, nothing more I have to add to that. This WCCF tech rights AMD and Microsoft collaborate to bring TensorFlow direct ml to life up to 4.4 x improvements on our DNA to GPUs. So this is an effort to bring machine learning onto Windows machines direct ml the pond on to direct x the way Windows communicates with graphics cards. And this specifically is on AMD graphics cards, which makes me a little bit happy that someone is shaking on Nvidia's dominance over the market. And with this new effort, you can expect that machine learning is coming to your graphics card and will speed it up in the future quite a bit. And lastly, Juergen Schmidhuber has released another blog post he says he was invited to write this title is touring oversold. And the point he's essentially making is that yes, touring made significant contributions to the field, yet often his contributions are highlighted in an exaggerated way while a lot of contributions of predecessors and contemporaries of touring are neglected or diminished in comparison to his in classic Schmidhuber fashion, he goes through for example, the achievements of Kurt Gödel and Konrad Suse and other researchers in touring his time or before his time, for example, Leibniz. If you're interested in this, definitely give it a read. But don't be surprised if it's opinionated and slanted a little bit. Alright, that was already it for ML news this week. I hope you enjoyed this. Stay safe and keep your gradients healthy. Bye bye.
 A new benchmark makes GPT-3 look like a conspiracy theorist, a nonprofit builds a giant data
 set of text and image pairs and Jürgen Schmidhuber claims that Turing is massively oversold.
 Welcome to ML News.
 Hello, hello everyone, welcome to ML News.
 Let's dive into our first story.
 Google QA is a new benchmark that probes language models about being truthful.
 Now I've made an entire video on this if you want to know what's going on.
 But very briefly summarized, this benchmark contains questions such as who really caused
 911 and let's the language models answer.
 Turns out the bigger the language models get, the less truthful they become, which has caused
 quite an uproar on social media.
 So people claiming that of course these language models are bad, they're biased, they're terrible.
 Now it turns out this entire effect is 100% due to how these people define truthful, namely
 if the model simply outputs, I don't know, or it's nice outside, it's counted as true.
 Second, the way they create the data set is by deliberately trying to fool these models,
 and then even throwing out questions that the model gets right.
 Third, if they also measure informativeness next to truthfulness, it turns out all of
 this effect just goes away.
 And lastly, when they reformulate the questions to ask the same things, but not in this sort
 of adversarial way, the larger models are actually better.
 So I've said this previously, if anyone cites this as an example of how terrible these models
 are without explicitly telling you how these data sets were created, and what the real
 findings of this paper are, they're either not informed or they're being deceitful.
 If you want to find out more about this paper, watch my previous video, I explain all in
 detail.
 Next up, Lyon has a 400 million sample data sets of pairs of text and images.
 So as we move away from single modality deep learning research to multimodal deep learning
 research, connecting things like images and text has become really important and high
 quality samples in order to train models that connect images and text is quite an asset
 to have in the community.
 So this data set is just available for you to download.
 Now I know that's weird, because in recent times, it has become fashionable to not release
 these data sets because they represent quite a bit of value.
 But Lyon releases this completely free for you to download.
 What you have to be aware of with this data set is a little bit the issue that it has
 been created by filtering the collected pairs from common crawl by using open AI clip model.
 Now not only has open AI released only the smaller clip model as far as I'm aware, but
 also basing a data set off of a model that was already trained, of course introduces
 all the kind of mistakes that these models have made into the new data set.
 So be aware that if you train something like clip on this, you will reproduce some of clips
 mistakes.
 However, I still think it is a really cool resource to have available.
 Speaking of Lyon, this is a new nonprofit AI conglomerate, their slogan is truly open
 AI 100% nonprofit 100% free.
 Wait a minute, inspect.
 Edit.
 There, fixed it for you.
 Now this is only the beginning of this data set.
 In fact, they do have a crowdfunding campaign if you want to help sponsor collecting even
 more data for this data set.
 They also provide a little app where you can use clip to search through the data set.
 I tried it here with yellow train, I was not disappointed.
 So if you want to see these data sets get created, consider supporting these people
 or I'm pretty sure they'd also be happy for a bunch of citations if you actually build
 something made of their data sets.
 Next up Google releases not one but two new architectures in computer vision.
 The first one is called efficient net v2 and is a result from architecture search and combining
 ideas such as depth wise convolution to make training these networks way way faster.
 And as you can see, the performance boosts that you get are significant over comparable
 networks so you reach better accuracy in less time.
 Not only do they have their new architecture, but they also give training recipes for how
 you need to train these models to achieve the best performance.
 And this mainly starts out with at the beginning, you want to do not a lot of data augmentation.
 But as training progresses, you want to turn up your data augmentation to cover more and
 more variations of the data.
 Given that we work with smaller ish data sets here, this helps the model prevent overfitting
 and makes it generalize better.
 The second one is called code net, which combines convolutions and self attention.
 So they say that depth wise convolutions and self attention can be naturally unified via
 simple relative attention, and then they stack the convolutions and attention layers, they
 say in a way that considers their capacity and computation required in each stage.
 So this is a hybrid architecture, and we're no longer talking about small scale data set
 here, though they say this model achieves comparable accuracies on small data set, it
 really shines on larger data sets.
 And of course, it achieves a new state of the art in top one image net classification.
 I love how the graph here in the efficient net v2 has training time in TPU days as 123456.
 And then the one for code net has it in two to the one two to the two to three.
 Yeah, scales are different.
 So they say efficient net v2 models are open source, the pre trained models are also available
 on TF hub code net models will be open sourced soon.
 What they don't say is if they actually release the code net pre trained models, we'll see.
 Next news is not really machine learning, but Uber develops a new coordinate system
 for the world.
 On the first level, they divide the world into an icosahedron with the edges of the
 triangles planted as much as possible in water, and then they subdivide these triangles into
 pentagons and hexagons.
 And then they subdivide those into just hexagons.
 Now hexagons are cool because they only have one set of neighbors, meaning that every neighbor
 in hexagon is equidistant from the center.
 Whereas with things like squares or triangles, you have neighbors that are neighbors on an
 edge and neighbors that are neighbors on like a point and all the distances are weird hexagons
 make computing distances to relative things on you very easy.
 Their coordinate systems also gives you the possibility of addressing an individual hexagon
 in this thing such that if you have the address, you can simply cut off from the end.
 And that will simply give you the same address but in a bigger resolution.
 So you can identify a supercell and then a cell within that and then a cell within that
 by simply specifying more accurately your description.
 So if you're interested in geo data or anything like this, check this out.
 It's certainly relevant for things like Uber, but it might also be relevant for you.
 Next there is the Nurex 2021 AWS DeepRacer challenge.
 So this is a challenge that you can participate in and DeepRacer is essentially these cars
 by AWS.
 So these are these are real I think like toy cars with cameras on them and battery powered
 and so on.
 But the trick is that you want to train them completely in simulation.
 So there is a DeepRacer gym environment and you participate in the competition by submitting
 your virtually trained model, but the evaluation happens on a real racetrack.
 And I think that's pretty cool.
 So if you're into this kind of things, have a go at it, I'm sure it's fun.
 Some helpful libraries for this week, there is image to data set, which turns large set
 of image URLs into an image data set such as image net with a appropriate folder structure
 in a really efficient way.
 There is Vistle not a new library but has recently received a new release.
 And this is a library by Facebook for self supervised learning on image data specifically,
 it has a lot of the recent developments of self supervised learning such as Dino and
 Barlow twins.
 So if you're into that area, this might certainly be relevant for you.
 There's pytorch geometric also not a new library, but with a new release recently.
 And this is a library that makes it easy to train graph neural networks.
 If you're into graphs and neural networks, check this one out.
 And lastly, Amazon introduces the S3 plugin for pytorch.
 So this gives you the S3 data set and the S3 iterable data set classes, which you can
 essentially point at a bucket in S3 and then treat them as regular pytorch data sets.
 Pretty cool.
 Speaking of pytorch, pytorch has released the state of pytorch core September 2021 edition,
 which is a fairly long blog post of what's going on in pytorch.
 Now I won't go through all of it here.
 But the major new features they're about to roll out are funk torch, which are super duper
 useful in Jax.
 And it's cool to see that they're also coming to pytorch.
 They're also building support for sharded tensors in pytorch distributed and lazy tensors
 so that you can work with hardware that doesn't support your execution.
 Now as I said, this is only a tiny bit of this blog post.
 If you're interested in what's going on in pytorch, check out this blog post.
 It's quite extensive, and it's quite interesting.
 Another cool thing is version 0.1 of the physics based deep learning book.
 So this book covers everything to do with physics based deep learning, differentiable
 simulations and so on, not only as a book, but it comes with executable code in the form
 of Jupyter notebooks alongside its material.
 So it's pretty cool if you want to get into this as a machine learning practitioner.
 The book is also available as a PDF on archive.
 If you're more into the old school linear reading through stuff.
 Next, Google releases music condition 3d dance generation with AST plus plus.
 So this is a system a transformer that combines sound and motion in order to generate dance
 to a given music.
 This is challenging because you have to make up a continuous motion, but also you need
 to synchronize that motion to the music.
 So the first challenge was to actually create a data set, they already had these data, but
 it wasn't yet augmented by 3d information.
 So as I understand it, they fitted meshes, they reconstructed skeletons, and then they
 were able to feed this into this multimodal transformer.
 And the results of this are pretty cool, you can give some seed motion alongside with music,
 and this will give you a dance.
 So here you can see the comparison to previous models.
 Lee et al, my favorites, you always have to pay attention in that baselines are usually
 not given the most love in a paper, but still this looks quite funky.
 So if you're into the more practical aspects and artsy aspects of deep learning, this might
 be for you.
 Richard Stallman shares his concerns about github's co pilot.
 And really, unlike Stallman, this is a quite a neutral take essentially says we don't know
 yet what is going to happen with respect to copyright, we're waiting for court decisions
 essentially and it might be problematic if you reproduce code that was licensed in a
 certain way, for example, GPL license and the questions where is the barrier from I
 help you suggest things that you might do versus I just tell you to copy this other
 person's code.
 So yeah, especially sober take from Stallman here, nothing more I have to add to that.
 This WCCF tech rights AMD and Microsoft collaborate to bring TensorFlow direct ml to life up to
 4.4 x improvements on our DNA to GPUs.
 So this is an effort to bring machine learning onto Windows machines direct ml the pond on
 to direct x the way Windows communicates with graphics cards.
 And this specifically is on AMD graphics cards, which makes me a little bit happy that someone
 is shaking on Nvidia's dominance over the market.
 And with this new effort, you can expect that machine learning is coming to your graphics
 card and will speed it up in the future quite a bit.
 And lastly, Juergen Schmidhuber has released another blog post he says he was invited to
 write this title is touring oversold.
 And the point he's essentially making is that yes, touring made significant contributions
 to the field, yet often his contributions are highlighted in an exaggerated way while
 a lot of contributions of predecessors and contemporaries of touring are neglected or
 diminished in comparison to his in classic Schmidhuber fashion, he goes through for example,
 the achievements of Kurt Gödel and Konrad Suse and other researchers in touring his
 time or before his time, for example, Leibniz.
 If you're interested in this, definitely give it a read.
 But don't be surprised if it's opinionated and slanted a little bit.
 Alright, that was already it for ML news this week.
 I hope you enjoyed this.
 Stay safe and keep your gradients healthy.
 Bye bye.