[ML News] GPT-4 Rumors | AI Mind Reading | Neuron Interaction Solved | AI Theorem Proving
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deep learning tutorial introduction to deep learning what is deep learning ml news mlnews kilcher news ai news gpt4 gpt 4 gpt 4 rumors gpt-4 mind reading ai mind reading mind reading machine learning machine learning news metagenomics byzantine bzyantine reviewer
#ai #mlnews #gpt4

Your weekly news from the AI & Machine Learning world.

OUTLINE:
0:00 - Introduction
0:25 - AI reads brain signals to predict what you're thinking
3:00 - Closed-form solution for neuron interactions
4:15 - GPT-4 rumors
6:50 - Cerebras supercomputer
7:45 - Meta releases metagenomics atlas
9:15 - AI advances in theorem proving
10:40 - Better diffusion models with expert denoisers
12:00 - BLOOMZ & mT0
13:05 - ICLR reviewers going mad
21:40 - Scaling Transformer inference
22:10 - Infinite nature flythrough generation
23:55 - Blazing fast denoising
24:45 - Large-scale AI training with MultiRay
25:30 - arXiv to include Hugging Face spaces
26:10 - Multilingual Diffusion
26:30 - Music source separation
26:50 - Multilingual CLIP
27:20 - Drug response prediction
27:50 - Helpful Things

ERRATA:
HF did not acquire spaces, they launched spaces themselves and supported Gradio from the start. They later acquired Gradio.

References:
AI reads brain signals to predict what you're thinking
https://mind-vis.github.io/?s=09&utm_source=pocket_saves
https://neurosciencenews.com/bmi-internal-speech-21837/

Closed-form solution for neuron interactions
https://twitter.com/ramin_m_h/status/1592585672606769153/photo/1
https://github.com/raminmh/CfC
https://github.com/raminmh/CfC/blob/main/torch_cfc.py

GPT-4 rumors
https://thealgorithmicbridge.substack.com/p/gpt-4-rumors-from-silicon-valley?utm_source=pocket_reader

Cerebras supercomputer
https://www.cerebras.net/andromeda/

Meta releases metagenomics atlas
https://ai.facebook.com/blog/protein-folding-esmfold-metagenomics/
https://www.genome.gov/genetics-glossary/Metagenomics

AI advances in theorem proving
https://ai.facebook.com/blog/ai-math-theorem-proving/
https://marketplace.visualstudio.com/items?itemName=jroesch.lean

Better diffusion models with expert denoisers
https://deepimagination.cc/eDiffi/

BLOOMZ & mT0
https://arxiv.org/abs/2211.01786?utm_source=pocket_reader
https://huggingface.co/bigscience/bloomz?text=Suggest+at+least+five+related+search+terms+to+%22M%E1%BA%A1ng+neural+nh%C3%A2n+t%E1%BA%A1o%22.

ICLR reviewers going mad
https://twitter.com/XiangruTang/status/1589703605098975237?utm_source=pocket_reader
https://twitter.com/BlancheMinerva/status/1588164585961422849?utm_source=pocket_reader
https://openreview.net/forum?id=pfuqQQCB34
https://twitter.com/peter_richtarik/status/1591408710366408706?utm_source=pocket_reader

Scaling Transformer inference
https://arxiv.org/abs/2211.05102

Infinite nature flythrough generation
https://ai.googleblog.com/2022/11/infinite-nature-generating-3d.html?utm_source=pocket_reader

Blazing fast denoising
https://github.com/dome272/Paella
https://arxiv.org/abs/2211.07292

Large-scale AI training with MultiRay
https://ai.facebook.com/blog/multiray-large-scale-AI-models/

arXiv to include Hugging Face spaces
https://blog.arxiv.org/2022/11/17/discover-state-of-the-art-machine-learning-demos-on-arxiv/

Multilingual Diffusion
https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltDiffusion

Music source separation
https://github.com/facebookresearch/demucs
https://arxiv.org/abs/2211.08553

Multilingual CLIP
https://twitter.com/rom1504/status/1593719037808320513

Drug response prediction
https://phys.org/news/2022-10-ai-accurately-human-response-drug.html

https://huggingface.co/Onodofthenorth/SD_PixelArt_SpriteSheet_Generator
https://huggingface.co/spaces/ronvolutional/sd-spritesheets
https://github.com/daspartho/prompt-extend
https://huggingface.co/blog/fine-tune-whisper
https://twitter.com/CarsonKatri/status/1585412662724272128
https://github.com/carson-katri/dream-textures/
https://www.youtube.com/playlist?list=PLzvYlJMoZ02Dxtwe-MmH4nOB5jYlMGBjr
https://github.com/xl0/lovely-tensors
https://github.com/jerryjliu/gpt_index
https://colab.research.google.com/drive/1o1qYJcFeywzCIdkfKJy7cTpgZTCM2EI4
https://dagshub.com/blog/launching-data-streaming-and-upload/
https://dagshub.com/blog/build-an-end-2-end-active-learning-pipeline-part-1/
https://github.com/run-ai/genv
https://arxiv.org/abs/2210.14868
https://github.com/timeseriesAI/tsai
https://medium.com/@yangyou_berkeley/diffusion-pretraining-and-hardware-fine-tuning-can-be-almost-7x-cheaper-85e970fe207b
https://medium.com/@hpcaitech/accelerating-structure-prediction-of-protein-monomers-and-multimer-by-11-times-769715dcb5b5
https://github.com/hpcaitech/ColossalAI/tree/main/examples/images/diffusion
https://arxiv.org/abs/2211.03726
https://github.com/Deci-AI/super-gradients
https://github.com/facebookresearch/shumai
https://github.com/huggingface/safetensors
https://github.com/google/learned_optimization/tree/main/learned_optimization/research/general_lopt
https://github.com/NVIDIA-Merlin/dataloader
https://loda-lang.org/
https://loda-lang.org/edit/
https://github.com/EelcoHoogendoorn/numga
https://arxiv.org/abs/2210.07316v1
https://huggingface.co/spaces/mteb/leaderboard
https://twitter.com/natfriedman/status/1575631194032549888
https://github.com/nat/natbot
 Rumors of GPT-4 are in the air, neuron transmissions is now solved in closed form, and mind reading is a thing now. It's Monday and welcome to ML News. Hello and welcome to ML News. This is your regular update of what's going on in the machine learning and AI world. Our first story is the most interesting one. Brain reading is more and more becoming a thing. There is a paper called Seeing Beyond the Brain conditional diffusion models with sparse masked modeling for vision decoding. In this paper, the authors give a visual stimulus to a subject, a real human and then look at their brain waves. This is non-invasive, this is fMRI brain scans. And from that reading of the fMRI, they're able to decode what the person is seeing. You can see right here on the top, you have visual stimuli. And on the bottom, you have the reconstructed images. Now what you'll be able to see is that the pixels don't exactly match. However, the semantic content is very often the same. Now this is done via aligning the latent spaces of the encoders for the brain data and encoders from images. And this has been a long standing problem because the training data that exists to map what people are seeing from their brain waves to the image space is just super sparse. But the authors here go around that by pre-training on unlabeled fMRI data and first get a very, very good autoencoder on that data going. Then the latent space can be determined, compressed. And then from that latent space, we can learn a conditional image diffusion decoder in order to map the visual stimuli to the encoding of the brain waves. So the paradigm that we see in deep learning where you want to do some unsupervised pre-training first, because you have much more unlabeled data and only then include the task specific data and learn that on top of the unsupervised pre-trained models also holds in the field of brain computer interfaces, apparently, that's pretty cool that we're more and more getting the chance to peek into people's brains. Now this isn't yet a full thought reader or anything like this. Essentially, they disambiguate between I believe some hundred different classes of labels, but it's still very, very cool that you can essentially reconstruct just from reading brain waves, what kind of image the person is seeing and what about is in the image in a related article, neuroscience news.com writes that brain machine interface device predicts internal speech. Now this is a little bit different in that it's actually invasive. So this is an interface directly to the brain, but it is able to predict internal speech, which means speech that you just internally think to yourself, it is able to decode that now it is not able to decode arbitrary speech, I believe they go up to about eight words or something like this. So it's not yet exactly super accurate, but we are making big, big progress in that front. Alright, next news. Ramin Hassani writes that they've published a new article in nature, machine intelligence and solved a differential equation that's been long standing without a closed form solution, we now have that closed form solution and it concerns the interactions between neurons. This is a major benefit for people who want to implement biologically inspired sort of biologically plausible neural networks, because previously, you'd have to have some sort of an ODE solver in order to even model that connection properly. And now that there's a closed form solution, you can essentially just forward and backprop through that formula. And the absolute coolest thing is that they have implemented this in both pytorch and TensorFlow. So you can technically build in this directly into your architectures today. Now it's not guaranteed to be like a lot better than what we currently have in terms of neuron neuron connection. But that's not the point. The point is to get to a place where we can simulate biologically plausible neural networks as well as possible. And from those potentially learn something about the brain and we might actually get some inspiration for how to even improve our artificial neural network architectures from this. So check out the paper and the repository in case you're interested. Alberto Romero on sub stack has an article called GPT for rumors from Silicon Valley. This is a summary of things that people whatever people means talk about currently around GPT for so open AI has been announcing like tiny bits of the next iteration of their language models here and there. And there used to be an interview by Sam Altman where he said GPT four isn't really going to be that much bigger than GPT three. And it's probably still going to be in the text domain, it's probably going to be a bit more aligned to humans a bit more, you know, learning from human feedback and so on. And people were kind of like a tiny bit disappointed, I guess, because it's not all we're going to build the next giant thing. But now more and more rumors are coming out that in fact, GPT four might be very well what they claim colossal. So another scale up of two orders of magnitude or something like this in terms of numbers of parameters or even three orders of magnitude, although some rumors claim that it is going to be sparse. So there's not really like a one to one comparison. On the other hand, there are also a lot of rumors that claim that GPT four is going to be multimodal after all. So including text, images, videos, and so on basically anything they can get their fingers on. So we'll see which one of these turns out to be true, it's very well possible that they first aim that just sort of improving GPT three and then all of a sudden with recent developments around diffusion models and so on, they've now gone into the direction of you know, let's just let's just do another giant leap. And from people who have apparently spoken to other people who have apparently tried the new model or a precursor to the new GPT four, they say that GPT four will be just as much an improvement over GPT three as GPT three was over GPT two. And if you remember in case you remember GPT three was a giant improvement over GPT two. Now is this going to be a GI and solve all our problems? Probably not. But in case this is true, in case it is really the same amount of step from GPT two to GPT three, as it is from GPT three to the new GPT four, then I think we're in for pretty pretty amazing times. In any case, rumors be rumors. And I guess we'll only know when we actually see it. The new model is rumored to be released sometimes between December and February. So the wait isn't going to be that long. Now related to this, OpenAI is also rumored to collaborate with Cerebros. And Cerebros in turn has just released their biggest supercomputer to date, which is called Andromeda has 13.5 million cores. Now Cerebros is a company that builds extremely large chips, they want to do as much as they can, like on a single chip. And that's why their chips are like, I think they're about yay big, I'm not exactly sure. But this absolute supercomputer is just comprised of 16 cerebros CS two systems. So that should give you an already an indication of just how big their individual systems already are now connecting them makes for a ginormous supercomputer. Now here on the website, it says get demo but I guess for most of you, it's not really going to be an option to go into business with this kind of scale. But for some of you, it might be and you might very well want to click that button. The meta research blog announces the ESM Metagenomic Atlas, the first view of the dark matter of the protein universe. So a lot of folding work a lot of protein folding work has been done recently with alpha fold and ESM fold and now meta releases a database of what's called meta genomics. Metagenomics is essentially if you just go outside and you pick up a piece of dirt, there's going to be like a ton of microbes, a ton of bacteria, a ton of organic material in there. And all of that genomic material isn't necessarily something you find in like the human genome project or something like this, yet it's still very important, for example, for ecology for medicine, but also for human well being. So this Metagenomic Atlas is the first database that reveals the structures of the meta genomic world at the scale of hundreds of millions of proteins, you can explore that there is a link to the Atlas right here. If you're anywhere near this world of protein folding, I guess this is a very exciting time. And I'm also excited for the progress we make on other frontiers rather than just scaling up and producing more stories about unicorns. Like for all the criticisms that these big models get and the pressure to just scale and scale and scale, they do every now and then deliver us something like this, something that's absolutely undeniably useful for some natural science out there. And as we get better with our core research, even if that's on pictures of cats, I strongly believe that this will greatly benefit adjacent fields such as biology, mathematics, physics, chemistry, and more of the other sciences. Also on the meta AI blog, they released a blog post called teaching AI advanced mathematical reasoning. Now I've dealt before with some of the papers that met I had in this regard, where they tried to come up with systems that use a prover. So there are these things called prover systems or proof assistance, or essentially formalize your whole mathematics inputs to spell out everything super formally, super descriptive, super detailed, and then you can use the system to search for new proofs by applying some proof strategies here and there. So you can say I want to do now a contra position of two things and so on. However, as you'll quickly discover the amount of strategies that you can apply to a given statement to search for a proof is really, really huge. And that leaves you essentially with a search problem. So this paper uses essentially a variant of Monte Carlo tree search, the same thing that like AlphaGo uses in order to determine the next moves in a go game in order to determine the next proof strategy or the next proof step that should be applied in order to reach a given target statement. Again, very cool that what initially dealt with a bunch of games and was really flashy because we can now solve go and chess much better has developed into something that is of actual use in an adjacent field in this case, mathematics. So very cool. Check out the paper if you are interested. Nvidia has released a paper called eDIF-i text to image diffusion models with ensemble of expert denoisers. This is I would say a typical Nvidia paper where they don't reinvent the world. But what they do is they take what exists and they apply a strong engineering mindset to it, they improve upon it, and it just results in a very high qualitative output. So in this case, they take the idea of these text to image diffusion models. But then on top of that, they have an ensemble of expert denoisers. So they don't just have one denoiser like we used to in a diffusion model, they have an ensemble of denoisers, which means that different models can take care of different phases in this denoising process. Also, they stage the image production in multiple steps. Now this has been done before, but it is a very viable strategy in that you essentially have one model produce a low resolution version of the image and then you successively scale that image. And then you successively scale that up. Now, as you can see right here, all in all that results in super high quality images that can either be done from a text description or from as you can see right here, text plus some kind of map or some kind of mask that you draw. Or over here, you can also input some sort of a style reference image into this system. So again, it's just amazing how people are able to push forward the state of the art in such a short time. Big Science has released two new models, one called blooms and the other one called empty zero. These are evolutions of their previous models. And they're mainly concerned with multitask prompted fine tuning. We've dealt with prompted fine tuning before in the galactica paper, which essentially means that after you retrain your model, you fine tune it on prompted samples. So like you would ask GPT three with a prompt to do some kind of task, you go ahead and actually fine tune on the prompt, the input and the output of that task to make the model learn to respond to such prompts in an appropriate fashion. And if you do that for multiple tasks, you also have the ability to then generalize to new tasks because that will carry over from the pre training. Specifically, these new models deal with this exact setting, but in non English data. So across lingual generalization doing this in multiple languages and potentially also generalizing across languages. The models are on hogging face if you want to check them out. I clear 2023 reviews are out on open review. And there are quite a few surprises in the negative direction. So Robert Tang here tweets out an example where the authors respond to a reviewer with response to you is a waste of time. I hope you can respect the author's work and give constructive comments instead of taking a few minutes to give a trivial suggestion. I recommend that you complete a university maybe kindergarten course before giving your review comments. That's just lovely. Somehow believing in the good of human beings, maybe this person just like had an absolutely terrible day and they really need this paper. And the review is actually very, very bad, like actually does make like a super trivial dunk on the paper. And you know, I'm not sure what happened right here. If you're ever inclined to write a rebuttal like this, just just sleep, go to sleep, wake up the next day, breathe and realize that it's it's kind of useless, even if it's probably true. Another worrying issue tweeted out by Stella Biederman is the following. So one reviewer criticized this model for that it is not acceptable to only compare with publicly available models, meaning that the paper should also have compared with non publicly available models. Now there is of course, a debate to have right here in order to properly compare to someone's model, you need to have access to it. On the other hand, there has been a long history of science where people just hadn't been putting stuff out into open source. And you'd essentially just have to take the numbers from the tables from their paper and then put those into your paper and essentially just believe what they said is possible that the reviewer here is of the stands that look, you know, you can just take the number that they claim and put them there. On the other hand, it's also entirely fair to say that, well, I don't have access to their model, I can't verify their numbers. And therefore, I'm not going to put them into my paper. The crux is obviously if that fact that you leave these things away that aren't public also makes your method appear a lot better in comparison because the only actual competitors to your method are closed source and only have some number in some paper. I don't know what's the correct answer right here. But it's certainly worth having a discussion about. And lastly, and you might actually have heard of this one is this paper called variance reduction is an antidote to Byzantine's better rates, weaker assumptions and communication compression as a cherry on the top. People do get creative with titles these days. But the problem that one reviewer here had is with the word Byzantine's, which the reviewer claimed to be disparaging of the whoever people consider themselves Byzantine. Now Byzantine is a term that's been long used in various fields of analysis, security, cryptography, I believe game theory. So the term is very well known and is an established technical term. However, the reviewer is of strong opinion that that is a term that contains prejudice and is derogatory and is denouncing the ethno religious practice of some people. Now the reviewer bases their opinion strongly on the fact that the ICLEAR code of ethics says you must respect cultural heritage of others and repeatedly claims that the usage of the term Byzantine in this work is a violation of the ICLEAR code of ethics. Whereas the authors claim this is a technical term, it's been used for a long time, and it is disparaging to absolutely no one. The conversation goes on and on, I believe there are over 36 comments in this thread, including some other people coming in and saying, hey, I'm actually considered Byzantine, and I don't have a problem with the term. So don't defend, you know us. Well, the reviewer did make some suggestions for other terms such as deviant. But the authors pointed out that none of these suggestions capture the term in its full existence or in how people actually use it. As the debate goes on, you'll see the reviewer shifting their stance a little bit from the fact that it's just not appropriate to use the term that the paper also isn't technically correct. But I strongly believe that the reviewers only introduced that point after the discussion had been going on for a while, and they realized they needed to make another stronger case on scientific terms. Now the problem here is that on open review, I believe you can't see the modifications. So we have no idea these comments, they were all changed around, even the original comment is changed around to like include some other feedback and so on. So it seems the timeline here is a little bit murky. The authors here also point out that this point, the point that the word Byzantine is inappropriate was apparently initially the only criticism of that reviewer or the only real criticism. But the reviewer gave the paper a really low score. And if you know anything about conferences, most meta reviewers just kind of look whether there is one bad score, and then the paper already has very poor chances or they look at the average, which would obviously be decreased strongly by one bad score. So essentially, the reviewer held the paper hostage a little bit and wanted the authors to change the wording. The authors even agree to abbreviate the word Byzantine to biz like the short form biz, because they just didn't agree that any of the other terms would do the technical nature justice. The reviewer disagreed that that would actually solve the problem and essentially said that even if they were to change the term, they would now expect not only to not use that term, but also the paper to contain a discussion of why the word Byzantine is not appropriate, or at least like a moral struggle of the authors are bringing this up of why this is problematic. The reviewer again, repeatedly and insistently claims that it violates the ICLR code of ethics and holds that as like a stick to like hit the authors with like code of ethics. This is against the code of ethics. What's interesting is that at some point, the program chairs commented on this as well, saying that the program chair committee and ethics chair have been following this thread closely upon preliminary investigation, the ethics chair find that the use of the B word, it's not the B word is a possibly emerging issue, but not yet a major ethics issue that could justify rejecting research, there seems to be no widespread agreement that the B word is offensive. This discussion between reviewers and authors is still valuable to our community, which raises awareness of this potentially emerging issue, we appreciate the thoughts from the reviews, and they said that this is essentially now resolved by saying, you know, reviewer, you made your point, but we don't agree with the point, the reviewer responded again, lengthily pointed out that this violates the ICLR code of ethics. Now in the end, you could say it's all good. And the program chairs came in and essentially squashed the reviewer and said, okay, the paper is fine, can use the word Byzantine, it's not problematic, all good. But I strongly actually believe that this is a big win for this reviewer right here, because the ethics chair, the appropriate response would be shut up, you're an embarrassment to the scientific institution, and you're barred from reviewing any more papers for any other conferences. This is a joke, shut up. But they didn't do that. They essentially said yes to the reviewer, they essentially said, yes, it's a possibly emerging issue, because they've seen that there was quite a bit of uproar in the community that such a what is essentially a technical term that is no one absolutely no one except this reviewer feels is not appropriate was used, the ethics chair said, yes, it's possibly emerging. So this is like a groundwork for the future. This is how these things slip in there, I have full conviction that people who write these codes of ethics do so with the best intentions, at least most of them, I do believe some of them predict exactly this. And this is how you again and again, slip these things in. So one person makes a fuss, you take the temperature of the community is like, not yet ready, but we have now precedence, right. So at the next conference, the same reviewer can make a fuss again, and they can point back and say, well, other people, you don't know, it's the same reviewer, other people have said this before. So actually, this might actually be problematic. And the ethics chair here seems to be bound by the fact that someone said this is ridiculous, shut up. However, they do so in the most lenient way in the most way that guarantees that in the future, this will actually become a problem. So in my opinion, big win for the reviewer right here, big win for the complainers, and I don't like it. Google has a new paper called efficiently scaling transformer inference on how they scale their big home models on TPUs. Now it is not going to be very applicable for most of you. But in case you care on how they enable something like 32 larger context lengths, and super duper flops and super duper hardware utilization during large batch processing, give this paper a read. Also from Google, the Google Research blog has an entry called infinite nature generating 3d fly throughs from still photos. This is on top of a paper that they published at ECCV, which generates infinite views or infinite fly throughs as the title says. And the cool thing is this happens from still images. So you can give a single image and it will generate a fly through from that image, they use various techniques for that. But the base idea is that you take an image and you predict its depth map. So how far away all the stuff is, and then you use that in order to render the image from a slightly different view. If you know how far away all the things are, you can position your camera slightly differently. And you can still determine where the pixels go. Now this will leave some pixels to be undetermined because you can now see behind things that you didn't see before. And then you have another model here in this refining step that essentially fills in these missing pixels. And then you repeat again, you pose the depth map, you adjust your camera position tiny bit, and then you fill in the pixels that are missing. In order to train this, it's not exactly super easy. But there are some various techniques called cycle consistency, or what they do right here, they have an adversarial setup, they have a discriminator to determine whether after a number of steps, the image still looks like it's been generated from a real like nature image. And if you back propagate that error, then you can generate very long, very high quality fly throughs through nature. Here you can see a bunch of examples. What I do find interesting is that they also added a specific sky model in order to make you feel like the sky is more real, I suspect their original works that the sky was often the problem and looked unrealistic. So now everything that sky here is produced actually by a separate model, as far as I can tell. Aya, I hope that's how you pronounce it is a new paper that also does text to image. However, this one is speed optimized. So in order to do diffusion, you have to take some bit of noise and then run it through the diffusion process step after step after step, there are various techniques to speed this up and paella supercharges them and manages to do the whole diffusion process in only 10 steps, which amounts to only 500 milliseconds. So within only 500 milliseconds, you have a high quality image from a given piece of text. Again, amazing progress in a field that is super young. Check out paella there is corresponding paper to it called fast text conditional discrete denoising on vector quantized latent spaces. Now, if you enjoyed the previous paper on how to scale up palm, then you might also enjoy multi ray, which is by meta, and the blog post is called optimizing efficiency for large scale AI models. This describes the system called multi ray, I've read the blog post. And I have to say it's kind of wishy washy, you have to guess a lot of the stuff, they just kind of describe in words what it does. And they link to various things that they've done. But I can't exactly read out, you know, what precisely they're doing right here. But if you need some inspiration of how a system like this would work, or you know, some hints of how this is really done in practice at scale, then give this blog post a read. Archive pairs up with hugging face. So previously, hugging face has acquired hugging face spaces from Gradio, which allows you to make little demos out of your hugging face repositories. And now archive includes those spaces. So if you upload a paper to archive, you can attach a demo from a hugging face space so people can directly on archive, try out your model if you have one or your technique if you have one and do so interactively. This is very cool. And obviously, I'm a big fan of integrating interactive things into our very old format of eight page PDFs. Okay, we've got a bunch of new models this week. The first one is alt diffusions by flag AI, which is a multi lingual diffusion model. So this is essentially stable diffusion, but multi lingual, as you can see right here, English, Chinese, Spanish, French, Russian, Japanese, Korean, Arabic, and Italian. Next is D mux by meta, which is a music source separation model. So this thing you can put like a song in there, and it will separate the sources meaning it will separate things like drums and vocals and isolate those perfect for practicing something doing karaoke, and whatever you want to do with it. The paper is called hybrid transformers for music source separation. And it's an archive, there's a new multi lingual clip available from lion trained on their own data set, the lion five B, and it reaches 77% zero shot on image net in English, and around 55% for Italian, Japanese and Chinese and supports over 100 languages. The cool thing is that it's very efficient in training because it uses locked image tuning, which we've discussed previously in a video. So check out the model and check out locked image tuning if you haven't seen it yet. It is really cool paper and cool and simple technique. In other news, a research group at the Citizen's University of New York has released a model that can accurately predict the human response to novel drug compounds. Now they're certainly not the first people to release such a model. This has obviously been going on for as long as data science has existed. But also, it's cool to see that even in this front in the drug discovery front, giant progress is being made on the back of what started out as cat image research. Alright, some helpful things for this week, we have quite a lot to get through. So let's get into it. This is a pixel art sprite sheet generator. If you're into old games into sprite animations, and so on. This is a stable diffusion based model that will create the sprites for you given a description. Look at this, I typed in fat Joey prompt extend is a model that will extend your prompts. So here is an example, you type in psychedelic liquids space, and it will append what it thinks that stable diffusion needs to give you what you want. So this is like a little bit of a translator between human input and whatever a very competent human using stable diffusion could do with all the modifiers such as concept art, sharp focus, illustration, Unreal Engine, and so on. There's a new blog post on hugging face telling you how to fine tune whisper for multilingual asr. But you can fine tune whisper for whatever you want. This blog post is your point of entry. dream texture is a plugin to make blender interact with stable diffusion. So here's a demo person types into blender, whatever they want as a texture in terms of text and then bada bing bada boom, apply, and it's now in the texture. Absolutely great. The YouTube channel mutual information has a series on reinforcement learning that I can highly recommend they spent a lot of time on this and I hope it is helpful to anyone who's looking to get into RL lovely tensors solves a problem we all have had in the past. So if I just want to print some tensor, I'm gonna get this and it's absolutely not helpful at all. As soon as your tensors are go beyond like four or five values, it's useless to just look at them. So all you do is you import lovely tensors, you monkey patch that stuff in and all of a sudden if you print a tensor, a non-py array, a torch tensor, whatever it will give you the shape, the amount of elements, statistics, the means the standard deviations and so on. This is a much, much better way to look at tensors. Now if the tensor is small enough, it will actually show you the values. But as soon as it's bigger than that, it will give you much more useful information. So here it warns you that there is infinities, there's nans in the tensors, and so on. And even here it tells you, well, this one is actually all zeros, you can still get back to the original tensor using sort of property access here, you have verbose access that will give you the values even if it's large. And here you get the just the plain old way if you really want that there are various helper methods around this also to show images to show statistics to show channels and to show things such as different filters in a stack of convolutional filters, I'll leave you to explore all of that yourself. But if you work with tensors a lot in an experimental sense, this is surely worth it. GPT index is a technique to build an index out of files using GPT. So this uses GPT to essentially take a bunch of files and then for example, recursively summarize them so that you essentially have a structure where you have a summary on top of a bunch of stuff. And then if you like one of them, you go into it and then you have summaries of the sub stuff that is there you go into that it's kind of an experimental I want to say this is a bit of a new way of thinking about what we could do with these models in order to organize information now that we have generative capabilities and I like that people think out of the box. So if you're also interested, check out this repository. There's a new upscaler for stable diffusion made by Rivers at Wings, the notebook is by N Shepherd and compute has been sponsored by stability AI. The notebook here runs you through the whole process of up sampling and it gives really cool results. I've previously talked about DAX hub DAX hub is like a bit of GitHub for machine learning. And I know a lot of places claim this nowadays, but that's how really believes in the open source paradigm. And now they release something they call direct data access and essentially a technique to stream down and upload version data to some place. So it essentially connects DVC, which you might know as like a data versioning tool with a transparent approach where you don't need to like pull all the whole data once or you know, stream it in some custom way, you can just treat it as if it already existed. And magically, the library in the background will pull down the data as you need it in a streamed fashion. So no long waiting on data to arrive, you can just simply like go train and even if you don't have space for the whole data will still work. Now I don't have exactly time here to explain you all of the things that you can do with it. But the install is really simple, essentially install their hooks and everything works just transparently and magically. So if you're interested, check it out and also check out their blog, it's regularly updated. For example, here is how to build an end to end active learning pipeline with fully open tools. GN is a GPU environment management tool lets you easily control, configure and monitor the GPU resources that you are using. And it is intended to ease up the process of GPU allocation for data scientists without code changes. So this is in case you're in some lab and you share GPUs with others, this tool is a must have I wish that this had existed during my PhD, it manages local GPUs, remote GPUs, cluster GPUs, and so on, you can reserve GPUs free up GPUs, essentially, whatever you want to do, it has even a VS code plugin. So if you're at all using GPUs, and especially if you're sharing them, consider this tool. MBXP is a multilingual benchmark for code completion in 10 plus programming languages. TSAI is an open source package intended for applying deep learning to time series on top of PyTorch and fast AI. Colossal AI has released two blog posts, both pertain to better and faster and cheaper training of models. The first one is what they call AI GC AI generated content, which essentially means image generation models. And the second one is for structure prediction of protein monomers and multimers. And both times they're able to speed up these models by a lot. Now the code is openly available. So do go and check it out. And the performance gains here are not only during inference, like we saw before, but this in fact provides for example, for stable diffusion, 6.5 times faster training and pre training cost savings. So the hardware cost of fine tuning can be almost seven times cheaper than if you were to do it in the vanilla way. HAP vid is a benchmark for tracking any point in a video. Super gradients is an awesome library to build, train and fine tune production ready deep learning state of the art vision models. Now we've seen a lot of libraries that you know, claim to just make stuff better. If you're into vision, I believe having like a library that's specific for vision, such as semantic segmentation or bounding box prediction, or even image classification, it really pays off to have a library that's dedicated to your field, especially if it's something like vision, where we have a lot of custom techniques that make these models just so much more efficient and better. But not only that super gradients also provides a lot of pre trained checkpoints. So even if you're just into using some models, this library might be good for you. Shumai is a network connected differentiable tensor library for TypeScript and JavaScript. As you can see in this demo, what you can do is you can define neural networks in TypeScript, and then you can distribute them over multiple places over multiple machines. And you can use the await like the async await syntax from JavaScript in order to ship data to other machines or call some function on another machine. And the library handles everything from you from forward propagation, even to back propagation and training. It's really cool. And the API for this looks quite clean. Safe tensors by hugging face is a new format who store and load tensors safely. I've previously done a video where I showed how you can like smuggle remote code execution into the hugging face hub, because the models essentially use the pytorch loading function. And pytorch in turn uses the pickle function by Python, which executes arbitrary code safe tensors is supposed to alleviate that by defining a safe, fixed and simple format to store tensors. Now, obviously, the trade off here is that you can't store arbitrary things anymore. If you want to store arbitrary things, you have to allow arbitrary code to be executed. So while I expect that a lot of architectures might switch to something like safe tensors, it is not a full solution for the problem. For better or worse, research will come up with new things, new ways of doing things. And if you constrain yourself to a particular way of doing things, then that will always not be enough. However, it's mostly going to be enough. Velo is a learn optimizer. And the cool thing here is that it really seems to be better than or at least on par with very hand tuned optimizers, you might know optimizers as stochastic gradient descent or Adam or something like this, but it is possible to learn an optimizer. So to learn a system that controls the optimization behavior of a training run of another system, these people have taken a lot of different ML problems, a lot of different networks have run optimization problems on them, and have essentially learned an optimizer that optimizes all of these different problems well. So that's what we consider a learned optimizer. And this one really seems that for many problems, especially like mainstream problems, it works really, really well out of the box. So without you having to tune, you know, the beta two parameters and the learning rate and stuff like this, you just apply it in its default configuration. And it does a pretty good job. This is super important if you want to do rapid prototyping, rapid exploration of some new ideas without doing a giant grid search over all the parameters. The Merlin data loader is a data loader specifically for recommender systems, recommender systems have, you know, a few extra or a few special requirements, namely, there's often quite few data I want to say compared to something like an image classifier, like the data points are mostly tabular, and they're not as many. So loading from disk and loading like hairs and stuff from this often can become the bottleneck. So a data loader is super important here. And the Merlin data loader promises to be over 10 times faster over native framework data loaders. If you're into recommender systems, try this out. Loda is an assembly language, a computational model, and a distributed tool for mining programs. This topic is very far away from me. But some of you might actually be interested. So if you're into integer sequences, there are these online encyclopedias of integer sequences, like 12345, and so on. So there's sequences of integers. And the question is always what's the program behind them? Like, can I come up with a piece of code that produces that integer sequence into perpetuity? And you know, 12345 is quite simple, but it gets complicated very quickly. And especially to teach machines to come up with the rules behind a sequence is a very challenging problem. So Loda is a system that allows you to mine such programs, essentially, you can run it and it will crank, crank, crank, crank and intelligently search for these programs. But not only that, it is also a distributed tool for doing that. So you can distribute you can partake in mining of such programs and much more. So as I understand, this is about what a loader program looks like or what it searches for. So here you can see one of these sequences. And this is apparently the program it comes up with. It looks pretty interesting. If you're interested, check Loda out. NumGa, not numba, numga is a library for geometric algebra in JAX and NumPy. If you're into geometric algebra, here's the example of a rigid body physics engine with a constraint solver, then this library might be for you. MTEB is a benchmark for text embedding. This is from similar authors as the buyer benchmark, which is a retrieval benchmark. But this goes further. This is a benchmark that covers eight embedding tasks over 56 data sets and 112 languages. And it also evaluates in this paper already 33 models on that benchmark. So the goal here is to find the one unified text embedding that covers all downstream tasks. And the status this far is that that universal embedding hasn't been found yet. The leaderboard shows that some models are good at some tasks, other models are good at other tasks. So the holy grail of text embedding is still somewhere out there. And this benchmark might prove that you have found it. Okay, the last cool thing I want to show you is Natbot. And this is already a little bit older, Nat Friedman tweeted this out in September, but essentially he managed to connect GPT-3 to the browser to a web browser and just let it interact with the web browser by prompting it in an appropriate way, given the website's HTML structure. So apparently the original idea comes from Sharif Shamim and that bot has a repository on GitHub. Look, it's just one Python file. I know half of you are super cringing right now. But yeah, research be research. And if you want to figure out how it's done, how that bot works. And if you want to give it a shot yourself might be really cool to do. So please do. Alright, that was all from ML news. This was a big chunk. Thank you so much for being here. Thank you for supporting the channel. Come to Discord if you're not already on it. Link is in the description. We have fantastic paper discussions every week and we talk general machine learning every day. With that being said, stay hydrated. Bye bye.
 Rumors of GPT-4 are in the air, neuron transmissions is now solved in closed form,
 and mind reading is a thing now. It's Monday and welcome to ML News.
 Hello and welcome to ML News. This is your regular update of what's going on in the machine learning
 and AI world. Our first story is the most interesting one. Brain reading is more and
 more becoming a thing. There is a paper called Seeing Beyond the Brain conditional diffusion
 models with sparse masked modeling for vision decoding. In this paper, the authors give a
 visual stimulus to a subject, a real human and then look at their brain waves. This is non-invasive,
 this is fMRI brain scans. And from that reading of the fMRI, they're able to decode what the person
 is seeing. You can see right here on the top, you have visual stimuli. And on the bottom, you have
 the reconstructed images. Now what you'll be able to see is that the pixels don't exactly match.
 However, the semantic content is very often the same. Now this is done via aligning the latent
 spaces of the encoders for the brain data and encoders from images. And this has been a long
 standing problem because the training data that exists to map what people are seeing from their
 brain waves to the image space is just super sparse. But the authors here go around that by
 pre-training on unlabeled fMRI data and first get a very, very good autoencoder on that data going.
 Then the latent space can be determined, compressed. And then from that latent space,
 we can learn a conditional image diffusion decoder in order to map the visual stimuli
 to the encoding of the brain waves. So the paradigm that we see in deep learning where you want to do
 some unsupervised pre-training first, because you have much more unlabeled data and only then
 include the task specific data and learn that on top of the unsupervised pre-trained models also
 holds in the field of brain computer interfaces, apparently, that's pretty cool that we're more and
 more getting the chance to peek into people's brains. Now this isn't yet a full thought reader
 or anything like this. Essentially, they disambiguate between I believe some hundred
 different classes of labels, but it's still very, very cool that you can essentially reconstruct
 just from reading brain waves, what kind of image the person is seeing and what about is in the image
 in a related article, neuroscience news.com writes that brain machine interface device predicts
 internal speech. Now this is a little bit different in that it's actually invasive. So this is an
 interface directly to the brain, but it is able to predict internal speech, which means speech that
 you just internally think to yourself, it is able to decode that now it is not able to decode arbitrary
 speech, I believe they go up to about eight words or something like this. So it's not yet exactly
 super accurate, but we are making big, big progress in that front. Alright, next news.
 Ramin Hassani writes that they've published a new article in nature, machine intelligence and
 solved a differential equation that's been long standing without a closed form solution, we now
 have that closed form solution and it concerns the interactions between neurons. This is a major
 benefit for people who want to implement biologically inspired sort of biologically
 plausible neural networks, because previously, you'd have to have some sort of an ODE solver in
 order to even model that connection properly. And now that there's a closed form solution,
 you can essentially just forward and backprop through that formula. And the absolute coolest thing
 is that they have implemented this in both pytorch and TensorFlow. So you can technically build in
 this directly into your architectures today. Now it's not guaranteed to be like a lot better than
 what we currently have in terms of neuron neuron connection. But that's not the point. The point
 is to get to a place where we can simulate biologically plausible neural networks as well
 as possible. And from those potentially learn something about the brain and we might actually
 get some inspiration for how to even improve our artificial neural network architectures from this.
 So check out the paper and the repository in case you're interested.
 Alberto Romero on sub stack has an article called GPT for rumors from Silicon Valley. This is a
 summary of things that people whatever people means talk about currently around GPT for so open AI has
 been announcing like tiny bits of the next iteration of their language models here and there.
 And there used to be an interview by Sam Altman where he said GPT four isn't really going to be
 that much bigger than GPT three. And it's probably still going to be in the text domain, it's probably
 going to be a bit more aligned to humans a bit more, you know, learning from human feedback and
 so on. And people were kind of like a tiny bit disappointed, I guess, because it's not all we're
 going to build the next giant thing. But now more and more rumors are coming out that in fact, GPT
 four might be very well what they claim colossal. So another scale up of two orders of magnitude or
 something like this in terms of numbers of parameters or even three orders of magnitude,
 although some rumors claim that it is going to be sparse. So there's not really like a one to one
 comparison. On the other hand, there are also a lot of rumors that claim that GPT four is going
 to be multimodal after all. So including text, images, videos, and so on basically anything they
 can get their fingers on. So we'll see which one of these turns out to be true, it's very well
 possible that they first aim that just sort of improving GPT three and then all of a sudden with
 recent developments around diffusion models and so on, they've now gone into the direction of you
 know, let's just let's just do another giant leap. And from people who have apparently spoken to
 other people who have apparently tried the new model or a precursor to the new GPT four, they
 say that GPT four will be just as much an improvement over GPT three as GPT three was over GPT two. And
 if you remember in case you remember GPT three was a giant improvement over GPT two. Now is this
 going to be a GI and solve all our problems? Probably not. But in case this is true, in case
 it is really the same amount of step from GPT two to GPT three, as it is from GPT three to the new
 GPT four, then I think we're in for pretty pretty amazing times. In any case, rumors be rumors. And
 I guess we'll only know when we actually see it. The new model is rumored to be released sometimes
 between December and February. So the wait isn't going to be that long. Now related to this,
 OpenAI is also rumored to collaborate with Cerebros. And Cerebros in turn has just released
 their biggest supercomputer to date, which is called Andromeda has 13.5 million cores. Now
 Cerebros is a company that builds extremely large chips, they want to do as much as they can, like
 on a single chip. And that's why their chips are like, I think they're about yay big, I'm not
 exactly sure. But this absolute supercomputer is just comprised of 16 cerebros CS two systems. So
 that should give you an already an indication of just how big their individual systems already are
 now connecting them makes for a ginormous supercomputer. Now here on the website, it
 says get demo but I guess for most of you, it's not really going to be an option to go into business
 with this kind of scale. But for some of you, it might be and you might very well want to click
 that button. The meta research blog announces the ESM Metagenomic Atlas, the first view of the dark
 matter of the protein universe. So a lot of folding work a lot of protein folding work has
 been done recently with alpha fold and ESM fold and now meta releases a database of what's called
 meta genomics. Metagenomics is essentially if you just go outside and you pick up a piece of dirt,
 there's going to be like a ton of microbes, a ton of bacteria, a ton of organic material in there.
 And all of that genomic material isn't necessarily something you find in like the human genome
 project or something like this, yet it's still very important, for example, for ecology for medicine,
 but also for human well being. So this Metagenomic Atlas is the first database that reveals the
 structures of the meta genomic world at the scale of hundreds of millions of proteins,
 you can explore that there is a link to the Atlas right here. If you're anywhere near this world of
 protein folding, I guess this is a very exciting time. And I'm also excited for the progress we
 make on other frontiers rather than just scaling up and producing more stories about unicorns.
 Like for all the criticisms that these big models get and the pressure to just scale and scale and
 scale, they do every now and then deliver us something like this, something that's absolutely
 undeniably useful for some natural science out there. And as we get better with our core research,
 even if that's on pictures of cats, I strongly believe that this will greatly benefit adjacent
 fields such as biology, mathematics, physics, chemistry, and more of the other sciences.
 Also on the meta AI blog, they released a blog post called teaching AI advanced mathematical
 reasoning. Now I've dealt before with some of the papers that met I had in this regard,
 where they tried to come up with systems that use a prover. So there are these things called prover
 systems or proof assistance, or essentially formalize your whole mathematics inputs to
 spell out everything super formally, super descriptive, super detailed, and then you can
 use the system to search for new proofs by applying some proof strategies here and there. So you can
 say I want to do now a contra position of two things and so on. However, as you'll quickly
 discover the amount of strategies that you can apply to a given statement to search for a proof
 is really, really huge. And that leaves you essentially with a search problem. So this paper
 uses essentially a variant of Monte Carlo tree search, the same thing that like AlphaGo uses
 in order to determine the next moves in a go game in order to determine the next proof strategy or
 the next proof step that should be applied in order to reach a given target statement. Again,
 very cool that what initially dealt with a bunch of games and was really flashy because we can now
 solve go and chess much better has developed into something that is of actual use in an adjacent
 field in this case, mathematics. So very cool. Check out the paper if you are interested.
 Nvidia has released a paper called eDIF-i text to image diffusion models with ensemble of expert
 denoisers. This is I would say a typical Nvidia paper where they don't reinvent the world. But
 what they do is they take what exists and they apply a strong engineering mindset to it, they
 improve upon it, and it just results in a very high qualitative output. So in this case, they take
 the idea of these text to image diffusion models. But then on top of that, they have an ensemble
 of expert denoisers. So they don't just have one denoiser like we used to in a diffusion model,
 they have an ensemble of denoisers, which means that different models can take care of different
 phases in this denoising process. Also, they stage the image production in multiple steps. Now this
 has been done before, but it is a very viable strategy in that you essentially have one model
 produce a low resolution version of the image and then you successively scale that image.
 And then you successively scale that up. Now, as you can see right here, all in all that results in
 super high quality images that can either be done from a text description or from as you can see
 right here, text plus some kind of map or some kind of mask that you draw. Or over here, you
 can also input some sort of a style reference image into this system. So again, it's just amazing
 how people are able to push forward the state of the art in such a short time. Big Science has
 released two new models, one called blooms and the other one called empty zero. These are evolutions
 of their previous models. And they're mainly concerned with multitask prompted fine tuning.
 We've dealt with prompted fine tuning before in the galactica paper, which essentially means that
 after you retrain your model, you fine tune it on prompted samples. So like you would ask GPT three
 with a prompt to do some kind of task, you go ahead and actually fine tune on the prompt, the input
 and the output of that task to make the model learn to respond to such prompts in an appropriate
 fashion. And if you do that for multiple tasks, you also have the ability to then generalize to
 new tasks because that will carry over from the pre training. Specifically, these new models
 deal with this exact setting, but in non English data. So across lingual generalization doing this
 in multiple languages and potentially also generalizing across languages. The models are
 on hogging face if you want to check them out. I clear 2023 reviews are out on open review. And
 there are quite a few surprises in the negative direction. So Robert Tang here tweets out an
 example where the authors respond to a reviewer with response to you is a waste of time. I hope
 you can respect the author's work and give constructive comments instead of taking a few
 minutes to give a trivial suggestion. I recommend that you complete a university maybe kindergarten
 course before giving your review comments. That's just lovely. Somehow believing in the good of
 human beings, maybe this person just like had an absolutely terrible day and they really need this
 paper. And the review is actually very, very bad, like actually does make like a super trivial dunk
 on the paper. And you know, I'm not sure what happened right here. If you're ever inclined
 to write a rebuttal like this, just just sleep, go to sleep, wake up the next day, breathe and
 realize that it's it's kind of useless, even if it's probably true. Another worrying issue tweeted
 out by Stella Biederman is the following. So one reviewer criticized this model for that it is not
 acceptable to only compare with publicly available models, meaning that the paper should also have
 compared with non publicly available models. Now there is of course, a debate to have right here
 in order to properly compare to someone's model, you need to have access to it. On the other hand,
 there has been a long history of science where people just hadn't been putting stuff out into
 open source. And you'd essentially just have to take the numbers from the tables from their paper
 and then put those into your paper and essentially just believe what they said is possible that the
 reviewer here is of the stands that look, you know, you can just take the number that they claim
 and put them there. On the other hand, it's also entirely fair to say that, well, I don't have
 access to their model, I can't verify their numbers. And therefore, I'm not going to put them
 into my paper. The crux is obviously if that fact that you leave these things away that aren't public
 also makes your method appear a lot better in comparison because the only actual competitors
 to your method are closed source and only have some number in some paper. I don't know what's
 the correct answer right here. But it's certainly worth having a discussion about. And lastly, and
 you might actually have heard of this one is this paper called variance reduction is an antidote to
 Byzantine's better rates, weaker assumptions and communication compression as a cherry on the top.
 People do get creative with titles these days. But the problem that one reviewer here had is with
 the word Byzantine's, which the reviewer claimed to be disparaging of the whoever people consider
 themselves Byzantine. Now Byzantine is a term that's been long used in various fields of analysis,
 security, cryptography, I believe game theory. So the term is very well known and is an established
 technical term. However, the reviewer is of strong opinion that that is a term that contains prejudice
 and is derogatory and is denouncing the ethno religious practice of some people. Now the
 reviewer bases their opinion strongly on the fact that the ICLEAR code of ethics says you must
 respect cultural heritage of others and repeatedly claims that the usage of the term Byzantine in
 this work is a violation of the ICLEAR code of ethics. Whereas the authors claim this is a
 technical term, it's been used for a long time, and it is disparaging to absolutely no one.
 The conversation goes on and on, I believe there are over 36 comments in this thread,
 including some other people coming in and saying, hey, I'm actually considered Byzantine,
 and I don't have a problem with the term. So don't defend, you know us. Well, the reviewer
 did make some suggestions for other terms such as deviant. But the authors pointed out that none of
 these suggestions capture the term in its full existence or in how people actually use it. As
 the debate goes on, you'll see the reviewer shifting their stance a little bit from the
 fact that it's just not appropriate to use the term that the paper also isn't technically correct.
 But I strongly believe that the reviewers only introduced that point after the discussion had
 been going on for a while, and they realized they needed to make another stronger case on
 scientific terms. Now the problem here is that on open review, I believe you can't see the
 modifications. So we have no idea these comments, they were all changed around, even the original
 comment is changed around to like include some other feedback and so on. So it seems the timeline
 here is a little bit murky. The authors here also point out that this point, the point that the word
 Byzantine is inappropriate was apparently initially the only criticism of that reviewer or the only
 real criticism. But the reviewer gave the paper a really low score. And if you know anything about
 conferences, most meta reviewers just kind of look whether there is one bad score, and then the paper
 already has very poor chances or they look at the average, which would obviously be decreased
 strongly by one bad score. So essentially, the reviewer held the paper hostage a little bit and
 wanted the authors to change the wording. The authors even agree to abbreviate the word Byzantine
 to biz like the short form biz, because they just didn't agree that any of the other terms would do
 the technical nature justice. The reviewer disagreed that that would actually solve the problem and
 essentially said that even if they were to change the term, they would now expect not only to not
 use that term, but also the paper to contain a discussion of why the word Byzantine is not
 appropriate, or at least like a moral struggle of the authors are bringing this up of why this is
 problematic. The reviewer again, repeatedly and insistently claims that it violates the
 ICLR code of ethics and holds that as like a stick to like hit the authors with like code of ethics.
 This is against the code of ethics. What's interesting is that at some point, the program
 chairs commented on this as well, saying that the program chair committee and ethics chair have been
 following this thread closely upon preliminary investigation, the ethics chair find that the use
 of the B word, it's not the B word is a possibly emerging issue, but not yet a major ethics issue
 that could justify rejecting research, there seems to be no widespread agreement that the B word is
 offensive. This discussion between reviewers and authors is still valuable to our community,
 which raises awareness of this potentially emerging issue, we appreciate the thoughts from the reviews,
 and they said that this is essentially now resolved by saying, you know, reviewer, you made your point,
 but we don't agree with the point, the reviewer responded again, lengthily pointed out that this
 violates the ICLR code of ethics. Now in the end, you could say it's all good. And the program chairs
 came in and essentially squashed the reviewer and said, okay, the paper is fine, can use the word
 Byzantine, it's not problematic, all good. But I strongly actually believe that this is a big win
 for this reviewer right here, because the ethics chair, the appropriate response would be shut up,
 you're an embarrassment to the scientific institution, and you're barred from reviewing
 any more papers for any other conferences. This is a joke, shut up. But they didn't do that. They
 essentially said yes to the reviewer, they essentially said, yes, it's a possibly emerging
 issue, because they've seen that there was quite a bit of uproar in the community that such a what
 is essentially a technical term that is no one absolutely no one except this reviewer feels is
 not appropriate was used, the ethics chair said, yes, it's possibly emerging. So this is like a
 groundwork for the future. This is how these things slip in there, I have full conviction that people
 who write these codes of ethics do so with the best intentions, at least most of them, I do believe
 some of them predict exactly this. And this is how you again and again, slip these things in. So
 one person makes a fuss, you take the temperature of the community is like, not yet ready, but we
 have now precedence, right. So at the next conference, the same reviewer can make a fuss
 again, and they can point back and say, well, other people, you don't know, it's the same reviewer,
 other people have said this before. So actually, this might actually be problematic. And the ethics
 chair here seems to be bound by the fact that someone said this is ridiculous, shut up. However,
 they do so in the most lenient way in the most way that guarantees that in the future, this will
 actually become a problem. So in my opinion, big win for the reviewer right here, big win for the
 complainers, and I don't like it. Google has a new paper called efficiently scaling transformer
 inference on how they scale their big home models on TPUs. Now it is not going to be very applicable
 for most of you. But in case you care on how they enable something like 32 larger context lengths,
 and super duper flops and super duper hardware utilization during large batch processing,
 give this paper a read. Also from Google, the Google Research blog has an entry called infinite
 nature generating 3d fly throughs from still photos. This is on top of a paper that they
 published at ECCV, which generates infinite views or infinite fly throughs as the title says. And
 the cool thing is this happens from still images. So you can give a single image and it will generate
 a fly through from that image, they use various techniques for that. But the base idea is that
 you take an image and you predict its depth map. So how far away all the stuff is, and then you use
 that in order to render the image from a slightly different view. If you know how far away all the
 things are, you can position your camera slightly differently. And you can still determine where the
 pixels go. Now this will leave some pixels to be undetermined because you can now see behind things
 that you didn't see before. And then you have another model here in this refining step that
 essentially fills in these missing pixels. And then you repeat again, you pose the depth map,
 you adjust your camera position tiny bit, and then you fill in the pixels that are missing. In order
 to train this, it's not exactly super easy. But there are some various techniques called cycle
 consistency, or what they do right here, they have an adversarial setup, they have a discriminator
 to determine whether after a number of steps, the image still looks like it's been generated from
 a real like nature image. And if you back propagate that error, then you can generate very long,
 very high quality fly throughs through nature. Here you can see a bunch of examples. What I do
 find interesting is that they also added a specific sky model in order to make you feel like the sky
 is more real, I suspect their original works that the sky was often the problem and looked
 unrealistic. So now everything that sky here is produced actually by a separate model, as far as
 I can tell. Aya, I hope that's how you pronounce it is a new paper that also does text
 to image. However, this one is speed optimized. So in order to do diffusion, you have to take some
 bit of noise and then run it through the diffusion process step after step after step, there are
 various techniques to speed this up and paella supercharges them and manages to do the whole
 diffusion process in only 10 steps, which amounts to only 500 milliseconds. So within only 500
 milliseconds, you have a high quality image from a given piece of text. Again, amazing progress in
 a field that is super young. Check out paella there is corresponding paper to it called fast
 text conditional discrete denoising on vector quantized latent spaces. Now, if you enjoyed
 the previous paper on how to scale up palm, then you might also enjoy multi ray, which is by meta,
 and the blog post is called optimizing efficiency for large scale AI models. This describes the
 system called multi ray, I've read the blog post. And I have to say it's kind of wishy washy, you
 have to guess a lot of the stuff, they just kind of describe in words what it does. And they link
 to various things that they've done. But I can't exactly read out, you know, what precisely they're
 doing right here. But if you need some inspiration of how a system like this would work, or you know,
 some hints of how this is really done in practice at scale, then give this blog post a read.
 Archive pairs up with hugging face. So previously, hugging face has acquired hugging face spaces from
 Gradio, which allows you to make little demos out of your hugging face repositories. And now
 archive includes those spaces. So if you upload a paper to archive, you can attach a demo from a
 hugging face space so people can directly on archive, try out your model if you have one or
 your technique if you have one and do so interactively. This is very cool. And obviously,
 I'm a big fan of integrating interactive things into our very old format of eight page PDFs.
 Okay, we've got a bunch of new models this week. The first one is alt diffusions by flag AI,
 which is a multi lingual diffusion model. So this is essentially stable diffusion, but multi lingual,
 as you can see right here, English, Chinese, Spanish, French, Russian, Japanese, Korean,
 Arabic, and Italian. Next is D mux by meta, which is a music source separation model. So this thing
 you can put like a song in there, and it will separate the sources meaning it will separate
 things like drums and vocals and isolate those perfect for practicing something doing karaoke,
 and whatever you want to do with it. The paper is called hybrid transformers for music source
 separation. And it's an archive, there's a new multi lingual clip available from lion trained on
 their own data set, the lion five B, and it reaches 77% zero shot on image net in English,
 and around 55% for Italian, Japanese and Chinese and supports over 100 languages. The cool thing
 is that it's very efficient in training because it uses locked image tuning, which we've discussed
 previously in a video. So check out the model and check out locked image tuning if you haven't seen
 it yet. It is really cool paper and cool and simple technique. In other news, a research group at the
 Citizen's University of New York has released a model that can accurately predict the human
 response to novel drug compounds. Now they're certainly not the first people to release such
 a model. This has obviously been going on for as long as data science has existed. But also,
 it's cool to see that even in this front in the drug discovery front, giant progress is being made
 on the back of what started out as cat image research. Alright, some helpful things for this
 week, we have quite a lot to get through. So let's get into it. This is a pixel art sprite sheet
 generator. If you're into old games into sprite animations, and so on. This is a stable diffusion
 based model that will create the sprites for you given a description. Look at this, I typed in fat
 Joey prompt extend is a model that will extend your prompts. So here is an example, you type in
 psychedelic liquids space, and it will append what it thinks that stable diffusion needs to give you
 what you want. So this is like a little bit of a translator between human input and whatever a very
 competent human using stable diffusion could do with all the modifiers such as concept art,
 sharp focus, illustration, Unreal Engine, and so on. There's a new blog post on hugging face telling
 you how to fine tune whisper for multilingual asr. But you can fine tune whisper for whatever you want.
 This blog post is your point of entry. dream texture is a plugin to make blender interact with
 stable diffusion. So here's a demo person types into blender, whatever they want as a texture in
 terms of text and then bada bing bada boom, apply, and it's now in the texture. Absolutely great.
 The YouTube channel mutual information has a series on reinforcement learning that I can
 highly recommend they spent a lot of time on this and I hope it is helpful to anyone who's looking
 to get into RL lovely tensors solves a problem we all have had in the past. So if I just want to
 print some tensor, I'm gonna get this and it's absolutely not helpful at all. As soon as your
 tensors are go beyond like four or five values, it's useless to just look at them. So all you do
 is you import lovely tensors, you monkey patch that stuff in and all of a sudden if you print
 a tensor, a non-py array, a torch tensor, whatever it will give you the shape, the amount of elements,
 statistics, the means the standard deviations and so on. This is a much, much better way to
 look at tensors. Now if the tensor is small enough, it will actually show you the values. But as soon
 as it's bigger than that, it will give you much more useful information. So here it warns you that
 there is infinities, there's nans in the tensors, and so on. And even here it tells you, well,
 this one is actually all zeros, you can still get back to the original tensor using sort of
 property access here, you have verbose access that will give you the values even if it's large. And
 here you get the just the plain old way if you really want that there are various helper methods
 around this also to show images to show statistics to show channels and to show things such as
 different filters in a stack of convolutional filters, I'll leave you to explore all of that
 yourself. But if you work with tensors a lot in an experimental sense, this is surely worth it.
 GPT index is a technique to build an index out of files using GPT. So this uses GPT to essentially
 take a bunch of files and then for example, recursively summarize them so that you essentially
 have a structure where you have a summary on top of a bunch of stuff. And then if you like one of
 them, you go into it and then you have summaries of the sub stuff that is there you go into that
 it's kind of an experimental I want to say this is a bit of a new way of thinking about what we
 could do with these models in order to organize information now that we have generative capabilities
 and I like that people think out of the box. So if you're also interested, check out this repository.
 There's a new upscaler for stable diffusion made by Rivers at Wings, the notebook is by
 N Shepherd and compute has been sponsored by stability AI. The notebook here runs you through
 the whole process of up sampling and it gives really cool results. I've previously talked about
 DAX hub DAX hub is like a bit of GitHub for machine learning. And I know a lot of places claim
 this nowadays, but that's how really believes in the open source paradigm. And now they release
 something they call direct data access and essentially a technique to stream down and
 upload version data to some place. So it essentially connects DVC, which you might know as like a data
 versioning tool with a transparent approach where you don't need to like pull all the whole data
 once or you know, stream it in some custom way, you can just treat it as if it already existed.
 And magically, the library in the background will pull down the data as you need it in a streamed
 fashion. So no long waiting on data to arrive, you can just simply like go train and even if you don't
 have space for the whole data will still work. Now I don't have exactly time here to explain you
 all of the things that you can do with it. But the install is really simple, essentially install
 their hooks and everything works just transparently and magically. So if you're interested, check it
 out and also check out their blog, it's regularly updated. For example, here is how to build an
 end to end active learning pipeline with fully open tools. GN is a GPU environment management tool
 lets you easily control, configure and monitor the GPU resources that you are using. And it is
 intended to ease up the process of GPU allocation for data scientists without code changes. So this
 is in case you're in some lab and you share GPUs with others, this tool is a must have I wish that
 this had existed during my PhD, it manages local GPUs, remote GPUs, cluster GPUs, and so on, you can
 reserve GPUs free up GPUs, essentially, whatever you want to do, it has even a VS code plugin. So
 if you're at all using GPUs, and especially if you're sharing them, consider this tool.
 MBXP is a multilingual benchmark for code completion in 10 plus programming languages.
 TSAI is an open source package intended for applying deep learning to time series on top
 of PyTorch and fast AI. Colossal AI has released two blog posts, both pertain to better and faster
 and cheaper training of models. The first one is what they call AI GC AI generated content,
 which essentially means image generation models. And the second one is for structure prediction
 of protein monomers and multimers. And both times they're able to speed up these models by a lot.
 Now the code is openly available. So do go and check it out. And the performance gains here are
 not only during inference, like we saw before, but this in fact provides for example, for stable
 diffusion, 6.5 times faster training and pre training cost savings. So the hardware cost of
 fine tuning can be almost seven times cheaper than if you were to do it in the vanilla way.
 HAP vid is a benchmark for tracking any point in a video. Super gradients is an awesome library
 to build, train and fine tune production ready deep learning state of the art vision models.
 Now we've seen a lot of libraries that you know, claim to just make stuff better. If you're into
 vision, I believe having like a library that's specific for vision, such as semantic segmentation
 or bounding box prediction, or even image classification, it really pays off to have
 a library that's dedicated to your field, especially if it's something like vision,
 where we have a lot of custom techniques that make these models just so much more efficient
 and better. But not only that super gradients also provides a lot of pre trained checkpoints. So even
 if you're just into using some models, this library might be good for you. Shumai is a network
 connected differentiable tensor library for TypeScript and JavaScript. As you can see in
 this demo, what you can do is you can define neural networks in TypeScript, and then you can
 distribute them over multiple places over multiple machines. And you can use the await like the
 async await syntax from JavaScript in order to ship data to other machines or call some function
 on another machine. And the library handles everything from you from forward propagation,
 even to back propagation and training. It's really cool. And the API for this looks quite clean.
 Safe tensors by hugging face is a new format who store and load tensors safely. I've previously
 done a video where I showed how you can like smuggle remote code execution into the hugging
 face hub, because the models essentially use the pytorch loading function. And pytorch in turn
 uses the pickle function by Python, which executes arbitrary code safe tensors is supposed to
 alleviate that by defining a safe, fixed and simple format to store tensors. Now, obviously,
 the trade off here is that you can't store arbitrary things anymore. If you want to store
 arbitrary things, you have to allow arbitrary code to be executed. So while I expect that a lot of
 architectures might switch to something like safe tensors, it is not a full solution for the problem.
 For better or worse, research will come up with new things, new ways of doing things. And if you
 constrain yourself to a particular way of doing things, then that will always not be enough.
 However, it's mostly going to be enough. Velo is a learn optimizer. And the cool thing here is that
 it really seems to be better than or at least on par with very hand tuned optimizers, you might
 know optimizers as stochastic gradient descent or Adam or something like this, but it is possible
 to learn an optimizer. So to learn a system that controls the optimization behavior of a training
 run of another system, these people have taken a lot of different ML problems, a lot of different
 networks have run optimization problems on them, and have essentially learned an optimizer that
 optimizes all of these different problems well. So that's what we consider a learned optimizer.
 And this one really seems that for many problems, especially like mainstream problems,
 it works really, really well out of the box. So without you having to tune, you know, the beta
 two parameters and the learning rate and stuff like this, you just apply it in its default
 configuration. And it does a pretty good job. This is super important if you want to do rapid
 prototyping, rapid exploration of some new ideas without doing a giant grid search over all the
 parameters. The Merlin data loader is a data loader specifically for recommender systems,
 recommender systems have, you know, a few extra or a few special requirements, namely, there's
 often quite few data I want to say compared to something like an image classifier, like the data
 points are mostly tabular, and they're not as many. So loading from disk and loading like hairs and
 stuff from this often can become the bottleneck. So a data loader is super important here. And the
 Merlin data loader promises to be over 10 times faster over native framework data loaders. If
 you're into recommender systems, try this out. Loda is an assembly language, a computational model,
 and a distributed tool for mining programs. This topic is very far away from me. But some of you
 might actually be interested. So if you're into integer sequences, there are these online
 encyclopedias of integer sequences, like 12345, and so on. So there's sequences of integers. And
 the question is always what's the program behind them? Like, can I come up with a piece of code
 that produces that integer sequence into perpetuity? And you know, 12345 is quite simple,
 but it gets complicated very quickly. And especially to teach machines to come up with the
 rules behind a sequence is a very challenging problem. So Loda is a system that allows you to
 mine such programs, essentially, you can run it and it will crank, crank, crank, crank and intelligently
 search for these programs. But not only that, it is also a distributed tool for doing that. So you
 can distribute you can partake in mining of such programs and much more. So as I understand, this
 is about what a loader program looks like or what it searches for. So here you can see one of these
 sequences. And this is apparently the program it comes up with. It looks pretty interesting.
 If you're interested, check Loda out. NumGa, not numba, numga is a library for geometric algebra
 in JAX and NumPy. If you're into geometric algebra, here's the example of a rigid body physics
 engine with a constraint solver, then this library might be for you. MTEB is a benchmark for text
 embedding. This is from similar authors as the buyer benchmark, which is a retrieval benchmark.
 But this goes further. This is a benchmark that covers eight embedding tasks over 56 data sets
 and 112 languages. And it also evaluates in this paper already 33 models on that benchmark. So the
 goal here is to find the one unified text embedding that covers all downstream tasks. And the status
 this far is that that universal embedding hasn't been found yet. The leaderboard shows that some
 models are good at some tasks, other models are good at other tasks. So the holy grail of text
 embedding is still somewhere out there. And this benchmark might prove that you have found it.
 Okay, the last cool thing I want to show you is Natbot. And this is already a little bit older,
 Nat Friedman tweeted this out in September, but essentially he managed to connect GPT-3 to the
 browser to a web browser and just let it interact with the web browser by prompting it in an
 appropriate way, given the website's HTML structure. So apparently the original idea comes from Sharif
 Shamim and that bot has a repository on GitHub. Look, it's just one Python file. I know half of
 you are super cringing right now. But yeah, research be research. And if you want to figure
 out how it's done, how that bot works. And if you want to give it a shot yourself might be really
 cool to do. So please do. Alright, that was all from ML news. This was a big chunk. Thank you so
 much for being here. Thank you for supporting the channel. Come to Discord if you're not already on
 it. Link is in the description. We have fantastic paper discussions every week and we talk general
 machine learning every day. With that being said, stay hydrated. Bye bye.