[ML News] Stanford HAI coins Foundation Models & High-profile case of plagiarism uncovered
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper plagiarism research plagiarism ml plagiarism foundation models tesla ai day comma three comma 3 george hotz elon musk stanford stanford ai stanford hai resnet momentum resnet lux ai neural mmo lex fridman dribnet clip pixelart pixelart ai art ai pixelart deep learning tutorial what is deep learning introduction to deep learning ml news mlnews
#plagiarism #foundationmodels #tesla

The best place to keep up to date with the latest and greatest from the ML world!

OUTLINE:
0:00 - Intro & Sponsor
3:15 - A high-profile case of plagiarism shocks the ML world
11:55 - Stanford AI releases paper on "Foundation Models"
19:45 - Updates on Apple's NeuralHash
20:45 - RL control for two-player splorts
21:45 - Tesla's AI Day
23:55 - COMMA THREE announced
24:40 - Intel winding down RealSense cameras
25:20 - IBM unveils Telum Processor
25:50 - Lux AI Challenge & Neural MMO Challenge
26:50 - Dribnet's CLIP PixelArt
27:40 - Multi-Agent RL papers are mostly fake
28:50 - I can't even come up with a segment title
29:25 - AI News Questions
31:20 - Frameworks & Libraries

Sponsor: Weights & Biases
https://wandb.ai

References:
Plagiarism case shocks ML world
https://arxiv.org/abs/2102.07870v1
https://arxiv.org/pdf/2102.07870v1.pdf
https://arxiv.org/abs/2108.05862
https://arxiv.org/pdf/2108.05862v1.pdf
https://www.reddit.com/r/MachineLearning/comments/p59pzp/d_imitation_is_the_sincerest_form_of_flattery/
https://michaelsdr.github.io/momentumnet/plagiarism/
https://www.zhihu.com/question/480075870/answer/2065820430?utm_source=pocket_mylist
https://zhuanlan.zhihu.com/p/400351960?utm_source=pocket_mylist
https://finance.sina.com.cn/tech/2021-08-17/doc-ikqciyzm1956801.shtml?utm_source=pocket_mylist
https://duoli.org/
https://web.archive.org/web/20210816025239/http://duoli.org/
https://twitter.com/shaohua0116/status/1427324015723487256/photo/1

Stanford AI targets Foundation Models
https://arxiv.org/abs/2108.07258
https://arxiv.org/pdf/2108.07258.pdf
https://ieeexplore.ieee.org/document/5206848
https://xgboost.readthedocs.io/en/latest/
https://en.wikipedia.org/wiki/Support-vector_machine
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
https://syncedreview.com/2019/06/27/the-staggering-cost-of-training-sota-ai-models/
https://openai.com/blog/better-language-models/

NeuralHash Saga Continues
https://www.reddit.com/r/MachineLearning/comments/p8q27o/p_run_neuralhash_in_your_browser/?utm_source=pocket_mylist
https://blog.roboflow.com/neuralhash-collision/
https://www.kron4.com/news/bay-area/bay-area-doctor-had-2000-child-pornography-images-and-videos-federal-complaint-alleges/

RL Control for competitive sports
https://ai.facebook.com/research/publications/control-strategies-for-physically-simulated-characters-performing-two-player-competitive-sports?utm_source=pocket_mylist

Tesla AI Day
https://www.youtube.com/watch?v=ABbDB6xri8o
https://spectrum.ieee.org/elon-musk-robot
https://www.youtube.com/watch?v=j0z4FweCy4M&t=4057s

George Hotz announces COMMA THREE
https://www.youtube.com/watch?v=jJn2OzOLIzo
https://comma.ai/shop/products/three

Intel abandons RealSense cameras
https://www.crn.com/news/components-peripherals/intel-says-it-s-winding-down-realsense-camera-business?itc=refresh

IBM unveils Telum Processor
https://www.prnewswire.com/news-releases/ibm-unveils-on-chip-accelerated-artificial-intelligence-processor-301360100.html

Kaggle Lux AI challenge
https://www.kaggle.com/c/lux-ai-2021

Neural MMO challenge
https://www.aicrowd.com/challenges/the-neural-mmo-challenge

Dribnet's PixelArt
https://twitter.com/dribnet/status/1426274645297094657

Multi-Agent RL papers mostly fake
https://www.reddit.com/r/reinforcementlearning/comments/p6g202/marl_top_conference_papers_are_ridiculous/

Elon Musk, Lex Fridman tweets trigger news story
https://www.benzinga.com/news/21/08/22610543/elon-musk-lex-fridman-see-language-evolving-with-help-of-artificial-intelligence

News Questions:
https://www.zdnet.com/article/can-ai-improve-your-pickup-lines/?utm_source=pocket_mylist
https://entertainment.inquirer.net/419318/what-if-the-simpsons-were-voiced-by-artificial-intelligence
https://www.analyticsinsight.net/which-career-should-you-choose-data-science-vs-artificial-intelligence/
https://www.bbc.co.uk/programmes/m000vl08?utm_source=pocket_mylist
https://ricochet.com/podcast/cosm-technology-summit/when-will-artificial-general-intelligence-actually-arise/
https://www.designnews.com/automation/how-smart-can-machine-get-check-out-new-artificial-intelligence
https://www.forbes.com/sites/anniebrown/2021/08/18/is-artificial-intelligence-contributing-positively-to-parenting-weighing-the-pros-and-cons-with-angela-j-kim/

3D Volleyball RL environment
https://www.reddit.com/r/MachineLearning/comments/p9aisc/p_a_3d_volleyball_reinforcement_learning/

Maze RL framework
https://enliteai.medium.com/maze-applied-reinforcement-learning-for-real-world-problems-e1ab6da1e167

Wanderer 2 HN Search
https://metaphor.so/
 high profile case of plagiarism shocks the machine learning world. Tesla has an AI day extravaganza and all of Stanford writes a single paper. Welcome to ML news. Stop! Before the rest of the video, this video is sponsored by Weights and Biases. Weights and Biases builds developer tools for machine learning for researchers for practitioners for juniors for seniors, whatever your favorite flavor of yogurt is, they don't care. They build products for you except cherry. Who likes cherry? Today, I want to talk to you about a feature called artifacts. So artifacts essentially are files in the cloud, but you're probably going to use them mostly for two things, data and models. Both of these things are notoriously tricky to work with data set is too large to check into get that we need to keep it up to date, we may have different versions of it and models even more, we want to save the outputs of our runs into models that we can then use later, maybe introspect. And these things are also versioned and we want to depend on them. So when I did this, I had to save the model to some special folder, and then I had to go grab it from that folder, put it on all the machines in a correct folder, and then reference that folder from all my scripts that would then consume this model with artifacts, this gets a lot easier. So we first uploaded the original data set to an artifact. Now we're going to consume that artifact, split the data into train validation and test data, and then emit those things as artifacts. So if there is a new version of the raw data available, I can simply run the same script depending on the same thing and it will create new versions of the train validation and test data, you can make this arbitrarily complex, but I hope you can see the point here. The same goes for models, if your run outputs and saves some kind of a model, you can log that as an artifact. And from then on, you can consume that model in all subsequent runs. Here's one of my models, it's a CNN, you can see it's already version 116 of that model. But you can see all I have to do to use this model in any code in any script in the future, I simply call the download method on the artifact and it will be available locally. And as I told you, you can do this with any file. But since this is a model of a deep learning framework, weights and biases understands it and gives me a neat viewer where I can actually introspect the model and look at the shapes and even at the weights of my CNN. So I think this is incredibly powerful. These things quickly get complicated with versions and scripts building upon other scripts. And the artifact framework really helps you to make sense of all of it. There's even the possibility that the data stays in specific private buckets with access controls. So not everyone in your team has access to all of the data. Of course, artifacts are only one of the features of weights and biases. If you're interested, please check them out. Free accounts are free. Academic accounts are free enterprise accounts cost a bit and that's it for this week's sponsor spot. Thanks a lot to weights and biases. Let's get into the video. So on a lonely August evening, I received the following text on Twitter, paper a plagiarized paper B and was accepted to ICCV. Now if you know anything about the academic world, especially the machine learning world is that everyone copies from everyone, but I gave the papers a look to confirm for myself. So here is paper a the first paper, the quote unquote original paper called momentum residual neural networks. It's a bunch of researchers of ENS, CNRS, and Google research. The basic idea is to bring some form of momentum to a residual neural network. Since a resnet resembles somewhat of an iterative process, the idea of momentum seems to be applicable here. The question is how exactly you do that. So here is a visualization of their idea. Formulas are here, there's lots of mathematical analysis, their experiments with these concentric rings and what happens to them. And there's like a table comparing it to previous approaches and so on. I'm looking at version one of the paper for anyone who's following jumping to the other paper, and I'm not going to reveal the name of the accused author right here because I don't want to point fingers at anything, I simply want to talk about the problem at hand. So the paper is called m revnet, deeper reversible neural networks with momentum that has quite a similar idea. In fact, there is a visualization of this flow, there are experiments with concentric rings being deformed, there is a neat little table comparing it to previous approaches. And generally the structure and even the sentences of entire passages appear to be just reformulations of one another at parts. Now I've looked further into this and realized that the first paper open source their code and the submission history reveals that they've probably tried to submit this to multiple conferences and failed a bunch of times before it got accepted. So the paper was out early hasn't been able to be published, code was out. And then the second paper appears. Now after looking at this carefully, I had the good impression that the second paper simply copied the first paper, ran their code with a bunch of different hyper parameters, maybe a different random seed and essentially wrote the same paper again, possibly hoping that they could get it through peer review before the first paper or that it would just be never be noticed at all. So I first told my discord community and contacted the authors, a bunch of people of my community also contacted the authors and got ahold of them, at which point they became aware and made the following statement on Twitter here, Abla says imitation is the sincerest form of flattery simply posting the two links, they followed up with a piece by piece comparison of the two papers essentially laying out a case of plagiarism. Now at this point, Twitter, Reddit and the different forums sprung into action looked into this, not only this, but also other papers, previous papers by the same author and dug up some worrisome conduct, but not only the Western world, but also the Chinese world. Now without revealing too much, the author in question happens to be studying at a Chinese university and working for Chinese companies. So the Chinese world sprung into action, comparing papers by this author and previous works and generally revealing this sort of approach to research where you take a paper and you do the visualizations in what is often actually a better way, but nevertheless, it's a copy. Now besides the first paper, there's a strong case for also a second paper being plagiarized. But that case is already very much more difficult. So people have pointed out things like similarities in formulas, similarities in the used signal pattern in the visualizations, and so on. In response to this, the co authors of that first author, as well as the supervisors quickly distanced themselves from the author saying they didn't know they weren't careful enough when looking at their work, they weren't that involved. And the first author responded by taking their personal homepage offline, though you can still access it via the internet archive and retracting the paper from archive with a comment given idea overlapped with existing work yet by the rules of archive, a retracted paper is still visible. If you simply go to v one of the paper, you can see the original version. The first author then went on social media and issued a somewhat apology saying that he made serious omissions by this and that he conducted the literature review for the paper before the other paper was out and didn't notice at the time of publication that the ideas overlap. In general, he tried to give an account of why the two papers are so similar and how this came about by just chance people having the same kinds of ideas and so on. Now safe to say this usually flies most cases of academic plagiarism, especially in machine learning are never ever caught or even pursued because you can always make the case well, it's a similar idea and so on and there are a bit different and whatnot. In this case, though, the case was so clear that I think the pressure was overwhelming. And the author edited the post to essentially say that they have plagiarized the two papers in question, they apologize, they will stop doing it, they will learn from it, and so on. Needless to say, this has generated a giant amounts of discussion. As I said, the Twitter post by Pierre Blanc became very widely spread, Reddit was on fire, Chinese social media talked about this at length, I was in general impressed with the amount of work that people put into analyzing similarities between papers. However, the best comment goes to a combination of this user right here, I don't know who it is, and Google Translate. It starts with after eating melon for a few days, you have already said a lot about this matter. I'm this is so cool. This is my this is my new go to saying I guess it's probably some sort of way to say after thinking about it for a few days or something like this. And it's a colloquial expression, but this is going to become my new go to sentence after eating melon for a few days, I've decided. Excellent, excellent. I love it. In addition to that, other people have come out with various stories of plagiarism, for example, Shah was on about code and papers that he reportedly only submitted to blind review, yet other papers have appeared that essentially are a copy of his work, which is even more shocking. It's not simply a person going on archive and pulling down publicly available information, not citing it, but essentially abusing their position as a anonymous peer reviewer. Now, as I said, the amount of things happening like this is uncountable, most of it will never ever get out or be done anything about it. The authors of the second paper here have retracted it from ICCV ICCV has already confirmed that this paper will not be published at ICCV and asked everyone to not call it the ICCV paper, which is why I dubbed it the paper formerly known as the ICCV paper. If you get this reference, you're old. So is this the end of the story? I don't know. As I said, plagiarism is still widespread, most of it goes on detected. And even from this particular author, it's very specific that he apologized for plagiarizing these two papers, people have pointed out similarities in other works and so on. And stemming from the fact that he first tried to simply go silent, then deny and now admitting to these two papers and combined with the fact that this author has had like a record number of papers in very short amount of time, it could be that this is simply a case of someone who let themselves be inspired by concurrent work a few times before and seeing how successful this is and not getting caught was getting more and more and more blunt in the plagiarism as time progressed. I can't state that for sure. I don't know, no one will ever be able to prove anything like this. So we'll just have to live with the fact that it is what it is. It goes on pretty much everywhere. I've personally witnessed quite a number of cases of people borrowing each other's ideas and even code. And what are you going to do? Nothing. Needless to say this isn't a case that we can solve easily with simple plagiarism checkers, which usually check for some sort of n gram overlap. And even if we have a sophisticated one, it's not going to help. As soon as people know that it exists, they're going to game it. So we'll have to live with this for the foreseeable future. There's a new paper called on the opportunities and risks of foundation models by everybody at Stanford. Every person has say in this. There are many authors to this paper, and it's sort of a position paper on what they call foundation models. Now, a few things, what it actually is, is mostly a literature review on what you might ask, well, foundation models, foundation models is this paper's framing of models that are kind of large and pre trained on large data and transfer learn then essentially think BERT GPT three clip, which they also state in the text, they say a foundation model is any model that is trained on broad data at scale and can be adapted to a wide range of downstream tasks. Now I have multiple problems with this 200 page monstrosity right here. The first one is with authorship itself, how do so many people work together on a single paper, the answer is they don't two people were sort of the integrators, and I guess the writers of the introduction and so on. And then the individual section of the papers were each authored by a subgroup of people, these subsections are even labeled with the individual authors and even contain things like joint first authorship of that subsection. Now in general, I'll say hey, it's a free world, do whatever you like. But this seems to be a little bit of a gaming of the citation system in academia, citations aren't weighted by number of authors or how much you contributed to anything, your names on there, you'll get a citation and this paper, ironically, might serve as sort of a foundation to be cited from many, many different other papers. Now you ask yourself the question, if someone wrote the section about adaptation of foundational models, should they really get a citation when someone is citing the section on misuse authored by a completely different set of authors? My personal opinion is no, this isn't a paper, this is a collection of papers like a compendium, a book, something like this. So it seems to be appropriate that when we cite this work, we cite the individual section of the work along with only the authors that wrote these individual sections. Now another problem that I and also other people have right here is that it's not really a new thing per se. Essentially, these people simply rebrand large pre trained models as foundation models. It's a very shaky definition. And it seems like it's just kind of a grab of a particular field or subfield for this particular group of people rather than simply contributing to the research landscape as a participant, there's a serious disconnect between the definition that they give for foundation models, a foundation model is any model that is trained on broad data at scale and can be adapted to a wide range of downstream tasks and what they actually talk about. Now generally in technical subjects, we do things such as we put up a definition of something and then we derive our conclusions, our experiments, our hypotheses and so on from that definition. However, this paper does something completely different. Essentially, none of the opportunities and risks they mentioned here are consequences of this definition. For example, a section on loss in accessibility. Why if foundation models are simply these models that can be adapted to things, how does that necessitate loss in accessibility? How does this necessarily impact the environment? I can see the large language models we have today do that. But how do you derive this from the definition like you can't? And how does the definition justify 200 pages? Essentially, if you amend the definition of foundation models to say something like there are efforts that cost a lot of money, and then a lot of other things are built upon these efforts, and that means anything that's built on top of it inherits all the properties, including all the problems, all the design decisions and so on all the properties of these intermediate efforts. And since it's costly to produce them, it's also costly to change them up their opportunity costs, their dangers of centralization of these things. And that that's about it. And that's with the extended definition. Now if you think about the definition, what comes to mind for me is something like a resonant 50, a pre trained resonant 50 on image net is used throughout the world is used in so many applications, a lot of people build on it, yet the number of people that actually fine tune GPT three outside of open AI is zero, the number of actual products that are built on in context learning is very limited. So if GPT three counts as a foundation model, the resonant 50 does after all it is a model trained on broad data at scale. Well, here is the paper on the image net data set large scale ergo. It's large scale and diversity ergo broad range. They say collecting image net is a challenging task. So not exactly cheap. They describe the data collection scheme and so on. And let's not forget the centrality and bias and data quality question in a resonant 50 image net the data set contains literal pornographic material. I've discussed this on my videos previously. So if resonant 50 doesn't count as a foundational model, then then I don't know how just because it's a few years old and doesn't cost as much as the models today, it fits every bit of the definition of a foundation model. Yeah, resonant 50 is mentioned one time in this 200 page document only to contrapose it to clip yet it's pretty clear what they actually mean GPT three, namely GPT three is mentioned over and over and over and over and over 65 times in this entire document only to be topped by Bert, which is mentioned a whopping 174 times, though sometimes it's like a sub part of another word. So rather than deriving conclusions from the definition, the paper is actually a series of anecdotes about some models that also fit the definition yet to me that doesn't justify the new term, especially if you go that far away from the definition. That's like me writing a paper on the opportunities and risks of group Ian models, which is any model containing an abelian group and I write 200 pages about how bad GPT three is because after all GPT three surely contains an abelian group somewhere in there. Now, with all the grumpiness I know it can get a bit much the paper is actually a great literature review on models such as GPT three, Dali clip, in general, the current models that are trained on large scale data and might not be entirely accessible to everyone. I'm not trying to deny that there are dangers to that. But let's keep in mind that for example, GPT two was also considered incredibly expensive and non accessible. And if you remember, even too dangerous to release at the point of release, yet these dangers haven't actually materialized. And as far as centralization of models go and choke points, I'm pretty sure it has happened previously in the machine learning world that pretty much everyone used the same couple of two or three really well working algorithms. No, can't think of any none of them. Well, okay, let's continue. So the community will have to decide if they accept this new term foundation models or if we just call GPT three and Bert by their names. Okay, next news, the neural hash story continues. There are now various projects in order to create collisions or run neural hash by itself. There's even one in the browser. I also have one if you want to watch the video. So also we have now reports that image net contains naturally occurring hash collisions by a robo flow here, you can search image net for things that elucidate the same neural hash, Apple has responded by saying that there is another server side check if to prevent wrong collisions and so on. But safe to say this neural hash system isn't the most effective you can evade it easily, you might be able to force collisions yet still we have a report from cron for that Bay Area doctor was found with 2000 images and videos of child pornography. We don't know exactly if this is already a result of this system. If it is, you know, good job works as intended that makes me happy that it worked here. It still doesn't make me more comfortable with the privacy implication of neural hash in general. Next news, Facebook AI research released a new paper called control strategies for physically simulated characters performing two player competitive sports. This is a reinforcement learning framework for control applications where you have mostly humanoids doing sports, but essentially the core parameters here are that there are a lot of degrees of freedom in some sort of a two player game in a continuous environment. I just love that the algorithm seems to come up with actual cool strategies and good control policies. It's not so easy for these things to balance themselves in the first place. And then to fight a boxing match where everyone tries to punch the other one to the ground is quite difficult. So you can see the difference between this new framework and sort of a comparison framework. I argue that the baseline though is the more interesting one, certainly. Oh, no. If you're interested in control and two player games, check it out. Tesla had its AI day. This was a big presentation where they talked about all their advancements into AI. I don't know if I should make an entire reaction video to that. I think I will. In the meantime, Lex Friedman has made an excellent overview over the most important things that happened there. I highly recommend you go check that out. And we have we have we have to talk about the Tesla bot. So the idea here is that all these technologies Tesla is developing for the car can also be deployed in a more general way in a humanoid robot to do manual labor. So this is from an article in IEEE spectrum. This is the slide that Tesla had up displaying the Tesla bot. Now besides the applications of eliminates dangerous, repetitive and boring tasks, it's also supposed to be friendly. Gotta gotta gotta love Elon Musk. Now needless to say, this is probably over promised both in whether or not that's doable at all with current or near future technology to the timeline they give, which is I think something like a year or so is probably not going to happen as advertised. But I come to think that Musk sometimes does things just to provoke exactly the reactions that we're getting. Elon Musk has no idea what he's doing with Tesla bot humanoid robots are way harder than Musk seems to think. Sometimes I wonder if he's like, what if I just tell them I'm going to build a robot in a year. Also, the way he introduced the robot is first, of course, it's just a mock up slides, but then he actually brought a human in a robot suit up on stage. And the human starts acting robotish, but then of course, increasingly gets less robotish. And you just see Elon smile back there. This was totally like you can imagine him sitting planning this out is like what if we like get a human and then just so the world decides whether this is funny or not. I think it's hilarious. This is 100% hilarious. As far as competitors go, George Hots revealed the comma three, which other than Tesla self driving approaches is a thing that you can put into a lot of different cars, essentially one mounted unit with cameras on it that is also supposed to do driving assistance. And I think something like fully self driving in the near future. There's also a big long presentation about the specs of the comma three, the problems with self driving with navigation in general with covering all of the edge cases and other than Tesla comma takes an open source approach where it actively wants the community of developers to help developing the product further. So if you are interested in that the comma three dev kit is available to order. Next news CRN writes Intel says it's winding down real sense camera business. So Intel was developing cameras, sensors and so on for computer vision application. Now it's saying it's shutting that down to focus on its core business. Middle of a loss if you had one of these or were planning on getting one of these, we've seen companies in the past saying they are going to focus on their core business. And it's not really clear what it means for some companies, it means they are on the edge of bankruptcy. While for others, it means they just want to make even more cash. Needless to say, if you're looking into sensors and vision hardware, Intel is no longer the place to do so. But IBM might be PR newswire writes IBM unveils on chip accelerated artificial intelligence processor. Okay, this is not a camera or a sensor. I just thought it was a great segue into the next segment. But IBM unveiled the Tulum processor, which essentially has an AI accelerator on chip. So a matrix multiplier, their idea is to bring the compute to where the data is and so on. But it's good to see a bit of competition in the market for accelerator chips. Okay, Kaggle has a new competition up called lux AI. This is essentially a two player game where you control units and have to collect as much light sources as possible to survive the night. So if you're interested in game playing agents give the lux AI challenge a try or if you are interested in game playing agents in very large world together with lots of other agents, look into AI crowds neural MMO challenge here you deploy an agent into a world with not just one other player, but many other players over longer periods of time. The goal is to collect resources and at the same time keep others from collecting their resources. It's very cool to see these kinds of challenges. You don't have to use reinforcement learning or anything, you can just script your bot if you want to. But it's usually cool to see which approaches win at the end in these very open world challenges. Very cool. Give it a try. Okay, at this point, I want to shout out to Dribnet who has been making a step into a bit of a different direction using the clip model and its image generation capabilities going into pixel art. And this looks very, very cool. So he's been generating various skylines and going through the ABC with various words zygote and zoo is Wellington, a yacht and a yakuza x ray and xenomorph. I love the idea that going to pixel art essentially blurs the line between human created and machine created even more. A lot of these pictures look absolutely fantastic. So this can be potentially used to just create funny pictures, but also can be combined, for example, to create video game assets and various other things where pixel art is generally used. Okay, following up a bit on the plagiarism issue, the reinforcement learning subreddit saw a big post saying that multi agent reinforcement learning top conference papers are ridiculous, essentially alleging that the entire field has a problem with unfair experimental tricks or cheating. Essentially, what you want to do is just implement really crappy baselines and then have your model be bigger, more powerful, take a longer time, have more information and do a better hyper parameter search essentially what we're used to from the entire field of machine learning, but the subfield of multi agent reinforcement learning because it's super noisy, and the experiments are mostly not standardized apparently has a particularly large problem with this. So there are people voicing in saying they've published in these fields. And this is absolutely true, mostly also that papers with solid experiments aren't getting published because I guess they're not as flashy as the paper with the tricked experiments. Needless to say, another bit of evidence that you shouldn't take the experimental results or any individual paper statements at face value. Benzinga writes, Elon Musk, Lex Friedman see language evolving with help of artificial intelligence. Wow, this sounds like a thing that they interview Elon Musk that they analyze years of work and integrated anything like this. No, no, they just they looked at they looked at two tweets, they looked at two tweets, and they made a news article about that. All right, AI helps a lot of people tweeting this right now, tweeting this right now. I want a news article tomorrow. You hear that tomorrow. Right now we come to our segment of AI news questions, which I answer absolutely without any context or reading the article. Here we go. ZD net writes, can AI improve your pickup lines? Wait, actually I need to write. Here's what comes up with Do you want to have a cup of coffee? Wow. You know, I guess for most people using pickup lines, simply saying please don't use pickup lines, just ask them for coffee is an improvement. So the answer is yes. The inquirer asks, what if the Simpsons were voiced by artificial intelligence? I don't care as long as Bart is still in Scientology. All is good. Presenza asks, artificial intelligence or human intelligence? I don't know. Probably depends on the tasks you want to solve. Analytics inside asks, which career should you choose data science versus artificial intelligence? Just learn the program, you'll be fine. Just learn the program. The BBC asks, is AI biased? Yes, the answer is yes, but probably not in the ways that the loudest people tell you. It's probably biased in a bit more of a boring way and probably a bit less in a oh my god, this is terrible way. Ricochet asks, when will artificial general intelligence actually arise to this technology summit here? I don't know. But neither do they. Design news asks, how smart can a machine get? I don't know. What's this question like seven smart machine can probably get seven smart. Cool. And Forbes asks, is artificial intelligence contributing positively to parenting? Let's check this out. Google what to do if my baby turns blue. If your baby is turning blue, calling 911 is very appropriate. Thanks AI. I guess the answer is yes. All right, that was it for our news questions. If you see a news question and want it answered without me reading anything, let me know. Okay, a few last shout outs. If you're old like me, you remember the good old days of blobby volley. Well, here's a 3d volleyball reinforcement learning environment built with Unity ML agents. Check it out. Also in light AI releases maze applied reinforcement learning for real world problems. It doesn't really have anything to do with an actual maze. It is yet another RL framework. But RL frameworks are kind of like there are many of them. And most of them have something wrong and something right. And if you haven't found any yet that fit you, maybe give this one a try. Lastly, metaphor releases wander to a large language model that was trained research through 2.5 million articles that were posted on hacker news. And yes, hacker news has a notoriously crappy search function. So thank you. Cool. This was it for this week's ML news. I thank you so much for checking in and checking out weights and biases. That being said, have a great rest of the week. I'll see you next Monday. Ciao.
 high profile case of plagiarism shocks the machine learning world. Tesla has an AI day
 extravaganza and all of Stanford writes a single paper. Welcome to ML news.
 Stop! Before the rest of the video, this video is sponsored by Weights and Biases. Weights
 and Biases builds developer tools for machine learning for researchers for practitioners
 for juniors for seniors, whatever your favorite flavor of yogurt is, they don't care. They
 build products for you except cherry. Who likes cherry? Today, I want to talk to you
 about a feature called artifacts. So artifacts essentially are files in the cloud, but you're
 probably going to use them mostly for two things, data and models. Both of these things
 are notoriously tricky to work with data set is too large to check into get that we need
 to keep it up to date, we may have different versions of it and models even more, we want
 to save the outputs of our runs into models that we can then use later, maybe introspect.
 And these things are also versioned and we want to depend on them. So when I did this,
 I had to save the model to some special folder, and then I had to go grab it from that folder,
 put it on all the machines in a correct folder, and then reference that folder from all my
 scripts that would then consume this model with artifacts, this gets a lot easier. So
 we first uploaded the original data set to an artifact. Now we're going to consume that
 artifact, split the data into train validation and test data, and then emit those things
 as artifacts. So if there is a new version of the raw data available, I can simply run
 the same script depending on the same thing and it will create new versions of the train
 validation and test data, you can make this arbitrarily complex, but I hope you can see
 the point here. The same goes for models, if your run outputs and saves some kind of
 a model, you can log that as an artifact. And from then on, you can consume that model
 in all subsequent runs. Here's one of my models, it's a CNN, you can see it's already version
 116 of that model. But you can see all I have to do to use this model in any code in any
 script in the future, I simply call the download method on the artifact and it will be available
 locally. And as I told you, you can do this with any file. But since this is a model of
 a deep learning framework, weights and biases understands it and gives me a neat viewer
 where I can actually introspect the model and look at the shapes and even at the weights
 of my CNN. So I think this is incredibly powerful. These things quickly get complicated with
 versions and scripts building upon other scripts. And the artifact framework really helps you
 to make sense of all of it. There's even the possibility that the data stays in specific
 private buckets with access controls. So not everyone in your team has access to all of
 the data. Of course, artifacts are only one of the features of weights and biases. If
 you're interested, please check them out. Free accounts are free. Academic accounts
 are free enterprise accounts cost a bit and that's it for this week's sponsor spot. Thanks
 a lot to weights and biases. Let's get into the video. So on a lonely August evening,
 I received the following text on Twitter, paper a plagiarized paper B and was accepted
 to ICCV. Now if you know anything about the academic world, especially the machine learning
 world is that everyone copies from everyone, but I gave the papers a look to confirm for
 myself. So here is paper a the first paper, the quote unquote original paper called momentum
 residual neural networks. It's a bunch of researchers of ENS, CNRS, and Google research.
 The basic idea is to bring some form of momentum to a residual neural network. Since a resnet
 resembles somewhat of an iterative process, the idea of momentum seems to be applicable
 here. The question is how exactly you do that. So here is a visualization of their idea.
 Formulas are here, there's lots of mathematical analysis, their experiments with these concentric
 rings and what happens to them. And there's like a table comparing it to previous approaches
 and so on. I'm looking at version one of the paper for anyone who's following jumping to
 the other paper, and I'm not going to reveal the name of the accused author right here
 because I don't want to point fingers at anything, I simply want to talk about the problem at
 hand. So the paper is called m revnet, deeper reversible neural networks with momentum that
 has quite a similar idea. In fact, there is a visualization of this flow, there are experiments
 with concentric rings being deformed, there is a neat little table comparing it to previous
 approaches. And generally the structure and even the sentences of entire passages appear
 to be just reformulations of one another at parts. Now I've looked further into this and
 realized that the first paper open source their code and the submission history reveals
 that they've probably tried to submit this to multiple conferences and failed a bunch
 of times before it got accepted. So the paper was out early hasn't been able to be published,
 code was out. And then the second paper appears. Now after looking at this carefully, I had
 the good impression that the second paper simply copied the first paper, ran their code
 with a bunch of different hyper parameters, maybe a different random seed and essentially
 wrote the same paper again, possibly hoping that they could get it through peer review
 before the first paper or that it would just be never be noticed at all. So I first told
 my discord community and contacted the authors, a bunch of people of my community also contacted
 the authors and got ahold of them, at which point they became aware and made the following
 statement on Twitter here, Abla says imitation is the sincerest form of flattery simply posting
 the two links, they followed up with a piece by piece comparison of the two papers essentially
 laying out a case of plagiarism. Now at this point, Twitter, Reddit and the different forums
 sprung into action looked into this, not only this, but also other papers, previous papers
 by the same author and dug up some worrisome conduct, but not only the Western world, but
 also the Chinese world. Now without revealing too much, the author in question happens to
 be studying at a Chinese university and working for Chinese companies. So the Chinese world
 sprung into action, comparing papers by this author and previous works and generally revealing
 this sort of approach to research where you take a paper and you do the visualizations
 in what is often actually a better way, but nevertheless, it's a copy. Now besides the
 first paper, there's a strong case for also a second paper being plagiarized. But that
 case is already very much more difficult. So people have pointed out things like similarities
 in formulas, similarities in the used signal pattern in the visualizations, and so on.
 In response to this, the co authors of that first author, as well as the supervisors quickly
 distanced themselves from the author saying they didn't know they weren't careful enough
 when looking at their work, they weren't that involved. And the first author responded by
 taking their personal homepage offline, though you can still access it via the internet archive
 and retracting the paper from archive with a comment given idea overlapped with existing
 work yet by the rules of archive, a retracted paper is still visible. If you simply go to
 v one of the paper, you can see the original version. The first author then went on social
 media and issued a somewhat apology saying that he made serious omissions by this and
 that he conducted the literature review for the paper before the other paper was out and
 didn't notice at the time of publication that the ideas overlap. In general, he tried to
 give an account of why the two papers are so similar and how this came about by just
 chance people having the same kinds of ideas and so on. Now safe to say this usually flies
 most cases of academic plagiarism, especially in machine learning are never ever caught
 or even pursued because you can always make the case well, it's a similar idea and so
 on and there are a bit different and whatnot. In this case, though, the case was so clear
 that I think the pressure was overwhelming. And the author edited the post to essentially
 say that they have plagiarized the two papers in question, they apologize, they will stop
 doing it, they will learn from it, and so on. Needless to say, this has generated a
 giant amounts of discussion. As I said, the Twitter post by Pierre Blanc became very widely
 spread, Reddit was on fire, Chinese social media talked about this at length, I was in
 general impressed with the amount of work that people put into analyzing similarities
 between papers. However, the best comment goes to a combination of this user right here,
 I don't know who it is, and Google Translate. It starts with after eating melon for a few
 days, you have already said a lot about this matter. I'm this is so cool. This is my this
 is my new go to saying I guess it's probably some sort of way to say after thinking about
 it for a few days or something like this. And it's a colloquial expression, but this
 is going to become my new go to sentence after eating melon for a few days, I've decided.
 Excellent, excellent. I love it. In addition to that, other people have come out with various
 stories of plagiarism, for example, Shah was on about code and papers that he reportedly
 only submitted to blind review, yet other papers have appeared that essentially are
 a copy of his work, which is even more shocking. It's not simply a person going on archive
 and pulling down publicly available information, not citing it, but essentially abusing their
 position as a anonymous peer reviewer. Now, as I said, the amount of things happening
 like this is uncountable, most of it will never ever get out or be done anything about
 it. The authors of the second paper here have retracted it from ICCV ICCV has already confirmed
 that this paper will not be published at ICCV and asked everyone to not call it the ICCV
 paper, which is why I dubbed it the paper formerly known as the ICCV paper. If you get
 this reference, you're old. So is this the end of the story? I don't know. As I said,
 plagiarism is still widespread, most of it goes on detected. And even from this particular
 author, it's very specific that he apologized for plagiarizing these two papers, people
 have pointed out similarities in other works and so on. And stemming from the fact that
 he first tried to simply go silent, then deny and now admitting to these two papers and
 combined with the fact that this author has had like a record number of papers in very
 short amount of time, it could be that this is simply a case of someone who let themselves
 be inspired by concurrent work a few times before and seeing how successful this is and
 not getting caught was getting more and more and more blunt in the plagiarism as time progressed.
 I can't state that for sure. I don't know, no one will ever be able to prove anything
 like this. So we'll just have to live with the fact that it is what it is. It goes on
 pretty much everywhere. I've personally witnessed quite a number of cases of people borrowing
 each other's ideas and even code. And what are you going to do? Nothing. Needless to
 say this isn't a case that we can solve easily with simple plagiarism checkers, which usually
 check for some sort of n gram overlap. And even if we have a sophisticated one, it's
 not going to help. As soon as people know that it exists, they're going to game it.
 So we'll have to live with this for the foreseeable future. There's a new paper called on the
 opportunities and risks of foundation models by everybody at Stanford. Every person has
 say in this. There are many authors to this paper, and it's sort of a position paper on
 what they call foundation models. Now, a few things, what it actually is, is mostly a literature
 review on what you might ask, well, foundation models, foundation models is this paper's
 framing of models that are kind of large and pre trained on large data and transfer learn
 then essentially think BERT GPT three clip, which they also state in the text, they say
 a foundation model is any model that is trained on broad data at scale and can be adapted
 to a wide range of downstream tasks. Now I have multiple problems with this 200 page monstrosity
 right here. The first one is with authorship itself, how do so many people work together
 on a single paper, the answer is they don't two people were sort of the integrators, and
 I guess the writers of the introduction and so on. And then the individual section of
 the papers were each authored by a subgroup of people, these subsections are even labeled
 with the individual authors and even contain things like joint first authorship of that
 subsection. Now in general, I'll say hey, it's a free world, do whatever you like. But
 this seems to be a little bit of a gaming of the citation system in academia, citations
 aren't weighted by number of authors or how much you contributed to anything, your names
 on there, you'll get a citation and this paper, ironically, might serve as sort of a foundation
 to be cited from many, many different other papers. Now you ask yourself the question,
 if someone wrote the section about adaptation of foundational models, should they really
 get a citation when someone is citing the section on misuse authored by a completely
 different set of authors? My personal opinion is no, this isn't a paper, this is a collection
 of papers like a compendium, a book, something like this. So it seems to be appropriate that
 when we cite this work, we cite the individual section of the work along with only the authors
 that wrote these individual sections. Now another problem that I and also other people
 have right here is that it's not really a new thing per se. Essentially, these people
 simply rebrand large pre trained models as foundation models. It's a very shaky definition.
 And it seems like it's just kind of a grab of a particular field or subfield for this
 particular group of people rather than simply contributing to the research landscape as
 a participant, there's a serious disconnect between the definition that they give for
 foundation models, a foundation model is any model that is trained on broad data at scale
 and can be adapted to a wide range of downstream tasks and what they actually talk about. Now
 generally in technical subjects, we do things such as we put up a definition of something
 and then we derive our conclusions, our experiments, our hypotheses and so on from that definition.
 However, this paper does something completely different. Essentially, none of the opportunities
 and risks they mentioned here are consequences of this definition. For example, a section
 on loss in accessibility. Why if foundation models are simply these models that can be
 adapted to things, how does that necessitate loss in accessibility? How does this necessarily
 impact the environment? I can see the large language models we have today do that. But
 how do you derive this from the definition like you can't? And how does the definition
 justify 200 pages? Essentially, if you amend the definition of foundation models to say
 something like there are efforts that cost a lot of money, and then a lot of other things
 are built upon these efforts, and that means anything that's built on top of it inherits
 all the properties, including all the problems, all the design decisions and so on all the
 properties of these intermediate efforts. And since it's costly to produce them, it's
 also costly to change them up their opportunity costs, their dangers of centralization of
 these things. And that that's about it. And that's with the extended definition. Now if
 you think about the definition, what comes to mind for me is something like a resonant
 50, a pre trained resonant 50 on image net is used throughout the world is used in so
 many applications, a lot of people build on it, yet the number of people that actually
 fine tune GPT three outside of open AI is zero, the number of actual products that are
 built on in context learning is very limited. So if GPT three counts as a foundation model,
 the resonant 50 does after all it is a model trained on broad data at scale. Well, here
 is the paper on the image net data set large scale ergo. It's large scale and diversity
 ergo broad range. They say collecting image net is a challenging task. So not exactly
 cheap. They describe the data collection scheme and so on. And let's not forget the centrality
 and bias and data quality question in a resonant 50 image net the data set contains literal
 pornographic material. I've discussed this on my videos previously. So if resonant 50
 doesn't count as a foundational model, then then I don't know how just because it's a
 few years old and doesn't cost as much as the models today, it fits every bit of the
 definition of a foundation model. Yeah, resonant 50 is mentioned one time in this 200 page
 document only to contrapose it to clip yet it's pretty clear what they actually mean
 GPT three, namely GPT three is mentioned over and over and over and over and over 65 times
 in this entire document only to be topped by Bert, which is mentioned a whopping 174
 times, though sometimes it's like a sub part of another word. So rather than deriving conclusions
 from the definition, the paper is actually a series of anecdotes about some models that
 also fit the definition yet to me that doesn't justify the new term, especially if you go
 that far away from the definition. That's like me writing a paper on the opportunities
 and risks of group Ian models, which is any model containing an abelian group and I write
 200 pages about how bad GPT three is because after all GPT three surely contains an abelian
 group somewhere in there. Now, with all the grumpiness I know it can get a bit much the
 paper is actually a great literature review on models such as GPT three, Dali clip, in
 general, the current models that are trained on large scale data and might not be entirely
 accessible to everyone. I'm not trying to deny that there are dangers to that. But let's
 keep in mind that for example, GPT two was also considered incredibly expensive and non
 accessible. And if you remember, even too dangerous to release at the point of release,
 yet these dangers haven't actually materialized. And as far as centralization of models go
 and choke points, I'm pretty sure it has happened previously in the machine learning world that
 pretty much everyone used the same couple of two or three really well working algorithms.
 No, can't think of any none of them. Well, okay, let's continue. So the community will
 have to decide if they accept this new term foundation models or if we just call GPT three
 and Bert by their names. Okay, next news, the neural hash story continues. There are
 now various projects in order to create collisions or run neural hash by itself. There's even
 one in the browser. I also have one if you want to watch the video. So also we have now
 reports that image net contains naturally occurring hash collisions by a robo flow here,
 you can search image net for things that elucidate the same neural hash, Apple has responded
 by saying that there is another server side check if to prevent wrong collisions and so
 on. But safe to say this neural hash system isn't the most effective you can evade it
 easily, you might be able to force collisions yet still we have a report from cron for that
 Bay Area doctor was found with 2000 images and videos of child pornography. We don't
 know exactly if this is already a result of this system. If it is, you know, good job
 works as intended that makes me happy that it worked here. It still doesn't make me more
 comfortable with the privacy implication of neural hash in general. Next news, Facebook
 AI research released a new paper called control strategies for physically simulated characters
 performing two player competitive sports. This is a reinforcement learning framework
 for control applications where you have mostly humanoids doing sports, but essentially the
 core parameters here are that there are a lot of degrees of freedom in some sort of
 a two player game in a continuous environment. I just love that the algorithm seems to come
 up with actual cool strategies and good control policies. It's not so easy for these things
 to balance themselves in the first place. And then to fight a boxing match where everyone
 tries to punch the other one to the ground is quite difficult. So you can see the difference
 between this new framework and sort of a comparison framework. I argue that the baseline though
 is the more interesting one, certainly. Oh, no. If you're interested in control and two
 player games, check it out. Tesla had its AI day. This was a big presentation where
 they talked about all their advancements into AI. I don't know if I should make an entire
 reaction video to that. I think I will. In the meantime, Lex Friedman has made an excellent
 overview over the most important things that happened there. I highly recommend you go
 check that out. And we have we have we have to talk about the Tesla bot. So the idea here
 is that all these technologies Tesla is developing for the car can also be deployed in a more
 general way in a humanoid robot to do manual labor. So this is from an article in IEEE
 spectrum. This is the slide that Tesla had up displaying the Tesla bot. Now besides the
 applications of eliminates dangerous, repetitive and boring tasks, it's also supposed to be
 friendly. Gotta gotta gotta love Elon Musk. Now needless to say, this is probably over
 promised both in whether or not that's doable at all with current or near future technology
 to the timeline they give, which is I think something like a year or so is probably not
 going to happen as advertised. But I come to think that Musk sometimes does things just
 to provoke exactly the reactions that we're getting. Elon Musk has no idea what he's doing
 with Tesla bot humanoid robots are way harder than Musk seems to think. Sometimes I wonder
 if he's like, what if I just tell them I'm going to build a robot in a year. Also, the
 way he introduced the robot is first, of course, it's just a mock up slides, but then he actually
 brought a human in a robot suit up on stage. And the human starts acting robotish, but
 then of course, increasingly gets less robotish. And you just see Elon smile back there. This
 was totally like you can imagine him sitting planning this out is like what if we like
 get a human and then just so the world decides whether this is funny or not. I think it's
 hilarious. This is 100% hilarious. As far as competitors go, George Hots revealed the
 comma three, which other than Tesla self driving approaches is a thing that you can put into
 a lot of different cars, essentially one mounted unit with cameras on it that is also supposed
 to do driving assistance. And I think something like fully self driving in the near future.
 There's also a big long presentation about the specs of the comma three, the problems
 with self driving with navigation in general with covering all of the edge cases and other
 than Tesla comma takes an open source approach where it actively wants the community of developers
 to help developing the product further. So if you are interested in that the comma three
 dev kit is available to order. Next news CRN writes Intel says it's winding down real sense
 camera business. So Intel was developing cameras, sensors and so on for computer vision application.
 Now it's saying it's shutting that down to focus on its core business. Middle of a loss
 if you had one of these or were planning on getting one of these, we've seen companies
 in the past saying they are going to focus on their core business. And it's not really
 clear what it means for some companies, it means they are on the edge of bankruptcy.
 While for others, it means they just want to make even more cash. Needless to say, if
 you're looking into sensors and vision hardware, Intel is no longer the place to do so. But
 IBM might be PR newswire writes IBM unveils on chip accelerated artificial intelligence
 processor. Okay, this is not a camera or a sensor. I just thought it was a great segue
 into the next segment. But IBM unveiled the Tulum processor, which essentially has an
 AI accelerator on chip. So a matrix multiplier, their idea is to bring the compute to where
 the data is and so on. But it's good to see a bit of competition in the market for accelerator
 chips. Okay, Kaggle has a new competition up called lux AI. This is essentially a two
 player game where you control units and have to collect as much light sources as possible
 to survive the night. So if you're interested in game playing agents give the lux AI challenge
 a try or if you are interested in game playing agents in very large world together with lots
 of other agents, look into AI crowds neural MMO challenge here you deploy an agent into
 a world with not just one other player, but many other players over longer periods of
 time. The goal is to collect resources and at the same time keep others from collecting
 their resources. It's very cool to see these kinds of challenges. You don't have to use
 reinforcement learning or anything, you can just script your bot if you want to. But it's
 usually cool to see which approaches win at the end in these very open world challenges.
 Very cool. Give it a try. Okay, at this point, I want to shout out to Dribnet who has been
 making a step into a bit of a different direction using the clip model and its image generation
 capabilities going into pixel art. And this looks very, very cool. So he's been generating
 various skylines and going through the ABC with various words zygote and zoo is Wellington,
 a yacht and a yakuza x ray and xenomorph. I love the idea that going to pixel art essentially
 blurs the line between human created and machine created even more. A lot of these pictures
 look absolutely fantastic. So this can be potentially used to just create funny pictures,
 but also can be combined, for example, to create video game assets and various other
 things where pixel art is generally used. Okay, following up a bit on the plagiarism
 issue, the reinforcement learning subreddit saw a big post saying that multi agent reinforcement
 learning top conference papers are ridiculous, essentially alleging that the entire field
 has a problem with unfair experimental tricks or cheating. Essentially, what you want to
 do is just implement really crappy baselines and then have your model be bigger, more powerful,
 take a longer time, have more information and do a better hyper parameter search essentially
 what we're used to from the entire field of machine learning, but the subfield of multi
 agent reinforcement learning because it's super noisy, and the experiments are mostly
 not standardized apparently has a particularly large problem with this. So there are people
 voicing in saying they've published in these fields. And this is absolutely true, mostly
 also that papers with solid experiments aren't getting published because I guess they're
 not as flashy as the paper with the tricked experiments. Needless to say, another bit
 of evidence that you shouldn't take the experimental results or any individual paper statements
 at face value. Benzinga writes, Elon Musk, Lex Friedman see language evolving with help
 of artificial intelligence. Wow, this sounds like a thing that they interview Elon Musk
 that they analyze years of work and integrated anything like this. No, no, they just they
 looked at they looked at two tweets, they looked at two tweets, and they made a news
 article about that. All right, AI helps a lot of people tweeting this right now, tweeting
 this right now. I want a news article tomorrow. You hear that tomorrow. Right now we come
 to our segment of AI news questions, which I answer absolutely without any context or
 reading the article. Here we go. ZD net writes, can AI improve your pickup lines? Wait, actually
 I need to write. Here's what comes up with Do you want to have a cup of coffee? Wow.
 You know, I guess for most people using pickup lines, simply saying please don't use pickup
 lines, just ask them for coffee is an improvement. So the answer is yes. The inquirer asks, what
 if the Simpsons were voiced by artificial intelligence? I don't care as long as Bart
 is still in Scientology. All is good. Presenza asks, artificial intelligence or human intelligence?
 I don't know. Probably depends on the tasks you want to solve. Analytics inside asks,
 which career should you choose data science versus artificial intelligence? Just learn
 the program, you'll be fine. Just learn the program. The BBC asks, is AI biased? Yes,
 the answer is yes, but probably not in the ways that the loudest people tell you. It's
 probably biased in a bit more of a boring way and probably a bit less in a oh my god,
 this is terrible way. Ricochet asks, when will artificial general intelligence actually
 arise to this technology summit here? I don't know. But neither do they. Design news asks,
 how smart can a machine get? I don't know. What's this question like seven smart machine
 can probably get seven smart. Cool. And Forbes asks, is artificial intelligence contributing
 positively to parenting? Let's check this out. Google what to do if my baby turns blue.
 If your baby is turning blue, calling 911 is very appropriate. Thanks AI. I guess the
 answer is yes. All right, that was it for our news questions. If you see a news question
 and want it answered without me reading anything, let me know. Okay, a few last shout outs.
 If you're old like me, you remember the good old days of blobby volley. Well, here's a
 3d volleyball reinforcement learning environment built with Unity ML agents. Check it out.
 Also in light AI releases maze applied reinforcement learning for real world problems. It doesn't
 really have anything to do with an actual maze. It is yet another RL framework. But
 RL frameworks are kind of like there are many of them. And most of them have something wrong
 and something right. And if you haven't found any yet that fit you, maybe give this one
 a try. Lastly, metaphor releases wander to a large language model that was trained research
 through 2.5 million articles that were posted on hacker news. And yes, hacker news has a
 notoriously crappy search function. So thank you. Cool. This was it for this week's ML
 news. I thank you so much for checking in and checking out weights and biases. That
 being said, have a great rest of the week. I'll see you next Monday. Ciao.