[Drama] Schmidhuber: Critique of Honda Prize for Dr. Hinton
deep learning machine learning schmidhuber hinton seppo rummelhardt hochreiter lstm rbm backpropagation credit science
Schmidhuber writes up a critique of Hinton receiving the Honda Price... AND HINTON REPLIES!

Schmidhuber's Blog Entry: http://people.idsia.ch/~juergen/critique-honda-prize-hinton.html
Hinton's Reply: https://www.reddit.com/r/MachineLearning/comments/g5ali0/d_schmidhuber_critique_of_honda_prize_for_dr/

Thumbnail Images:
By Eviatar Bach -https://de.m.wikipedia.org/wiki/Datei:Geoffrey_Hinton_at_UBC.jpg
By ITU/R.Farrell - https://www.flickr.com/photos/itupictures/34343385563, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=75018240

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 On April 21st, Jürgen Schmidhuber tweeted out, Stop crediting the wrong people for inventions made by others. At least in science, the facts will always win at the end, as long as the facts have not yet won. It is not yet the end. No fancy award can ever change that. Hashtag self-correcting science, hashtag plagiarism. And links to an article of his own website, where he wrote, Critique of Honda Prize for Dr. Hinton. So this is on Schmidhuber's own website, and it's by himself. Don't you love this? How to pronounce his name, Jürgen Schmidhuber. You again. Sorry. This is absolutely great. So both, actually, Schmidhuber and Hinton are on Twitter. You can tweet at them and follow them. This article here is basically a critique of the press release of Honda when they awarded Jeff Hinton for his achievements. And it goes through it step by step. And we won't look at the whole thing, but just for you to get the flavor. So here Honda says, Dr. Hinton has created a number of technologies that have enabled the broader application of AI, including the backpropagation algorithm that forms the basis of deep learning approach to AI. And Schmidhuber just goes off. He basically claims, while Hinton and his coworkers have made certain significant contributions to deep learning, the claim above is plain wrong. He says Hinton did not invent backpropagation. The person who invented backpropagation was Seppo Linayma. He says basically many papers failed to cite Linayma, who was the original inventor of backprop and so on. And he goes through a history of this and how it's even earlier. I always have a bit of a trouble with claims like who invented what, because when is an algorithm really the same thing? And when is it a variation on another algorithm? And when is it something completely new? It's never entirely clear. But the points here made that the things, the backpropagation algorithm existed before Hinton. And also that some of the papers, some of the seminal papers did not cite the correct origin. Statement 2. In 2002 he introduced a fast learning algorithm for restricted Boltzmann machines that allowed them to learn a single layer of distributed representation without requiring any labeled data. These methods allowed deep learning to work better and they led to the current deep learning revolution. And he basically goes, no, Dr. Hinton's interesting unsupervised pre-training for deep neural networks was irrelevant for the current deep learning revolution. In 2010 our team showed that the feed forward networks can be trained by plain backprop, do not at all require pre-training. And he basically again says, apart from this Hinton's unsupervised pre-training was conceptually a rehash of my unsupervised pre-training for deep recurrent neural networks. So he, you know, as you know, Schmidhuber has done a lot of work in recurrent neural networks and he basically says it was just a rehash of his algorithm. Now I have to say I have, so first of all, he makes a point here, right, that we don't really do unsupervised pre-training anymore until now, of course. But you, like, to train an MNIST classifier, you don't have to do that. But it's also doubtful that this was a step, even though if it wasn't on the exact path to the current situation, it was a thing that got people excited maybe. And so the critique is like half valid. And also it doesn't help Schmidhuber that he always compares it to his own things. Like it just, like, either criticize them for, you know, general things, but then avoid bringing your own things in because it just sounds like I did this before. And also I read some papers from these times. People just wrote papers sometimes. I haven't read this specific one, but sometimes people just wrote papers writing down their ideas. Like, one could do this and this and this. Never doing any experiments or actually specifying exactly what they mean. They just kind of wrote down a bunch of ideas and that got published. Especially, like, there's some reinforcement learning papers where people are just like, oh, one, I imagine agents doing this and learning from that. So it is, again, it is never really clear. Ideas are just had by everyone. I think people mistake this, that think that ideas are unique. It's not ideas that are unique. Many people have the same ideas, but some... There's also execution and exact formalization and so on. And exact level of specificity. All of this is really hard. And then the Honda says, in 2009, Dr. Hinton and two of his students used multilayer neural nets to make major breakthrough in speech recognition. That led directly to greatly improved. And this, of course, Schmidrub goes off by this because speech recognition is, of course, prime LSTM territory. So you don't want to go near this. And the Honda further says, revolutionized computer vision by showing that deep learning worked far better than existing state of the art. And again, he says the basic ingredients were already there and so on. And our team in Switzerland already used his first superior award-winning GPU-based CNN and so on. That's what was called DanNet, was produced by his group. And again, this seems correct, right? This seems when he lays it out like this, it doesn't change the fact that AlexNet won ImageNet in 2012. And that was like the start of the deep learning revolution. It was like, wow, you can cut the error rate by something like 30% simply by doing this deep learning stuff. So again, even if DanNet says it blew away the competition, it always seems like Schmidhuber is kind of right, but then also he's not. He's like, exact academic work and the idea being there on a paper isn't the only thing that drives progress. And it says, to achieve their dramatic results, Dr. Hinton also invented a widely used new method called dropout, which uses overfitting. No, like no, like no, just no. Randomly dropping parts in order to make something more robust, that is surely not a new thing. And he also says much earlier, there's this stochastic delta rule and so on. And he also critiques that this paper did not cite this. They just gave it the name. This is an idea that is kind of so simple that you wouldn't even necessarily think about researching whether that has existed already. I think they just did it because it's a natural idea, and then they gave it a name, and the name stuck. It's not about the idea itself. And then lastly, they say, of the countless AI-based technological services across the world, it is no exaggeration to say that few would have been possible without the results Dr. Hinton created. I love this. Name one that would not have been possible. And he just gives a list of their own group that are basically possible without Hinton's contributions. And this is just a bit of a cheap shot. Clearly, Honda, they're not saying it would have been physically impossible. Without his contributions. But certainly Hinton has, even if he hadn't invented any of those things, he certainly has created a spark. And these things created a splash, got people excited, people thinking about new ways of applying things, even if this is all true. But I would like you to notice, this is a critique of what Honda says about Hinton. And if I read through the statements of Schmidhuber, most of them are technically correct. And you know, that was that. And then I thought, OK, cool. But then someone posted it on Reddit. And then Hinton replies. And this is... Don't you love this? So Hinton says, Having a public debate with Schmidhuber about academic credit is not advisable because it just encourages him. And there is no limit to the time and effort that he is willing to put into trying to discredit his perceived arrivals. He is even escorted to tricks like having multiple aliases in Wikipedia to make it look as if other people agree. The page on his website about Alan Turing is a nice example of how he goes on trying to... These are shots fired. And he says, I'm going to respond once and only once. I have never claimed that I invented backpropagation. David Rumelhard invented it independently. After other people in other fields had invented it. It's true. When we first published, we did not know the history. So he basically says, OK, we did forget to cite it when we first published about backprop, but he doesn't say he invented it. What I've claimed is that I was the person to clearly demonstrate that backprop could learn interesting internal representations and that this is what made it popular. So this goes into the direction. Schmidhuber is very much on academic contributions. Idea was there before. And Hinton basically says, no, what we did is kind of we showed that it works in this particular way and we kind of got people excited about it. I did this by forcing that, blah, blah, blah. And he says, it is true that many people in the press have said I invented backprop and I've spent a lot of time correcting them. Here is an excerpt from 2018 where this is, I guess, a quote from this book that quotes Hinton where he says, lots of people invented different versions of backprop before David Rumelhart. They were mainly independent inventions, something I feel I've got too much credit for. It's one of these rare cases where an academic feels he has gotten too much credit for something. My main contribution was to show you can use it for learning distributor representations. So I'd like to set the record straight on that. Then he says, maybe Juergen would like to set the record straight on who invented LSTMs. Boom, boom. Crazy. Shots fired by Hinton here. This is, I mean, this is just great. But again, look at what Hinton says. Hinton basically says, yes, I have not invented that. I have corrected this on public record in the past. And yeah, so that's what Hinton says. And I mean, the comments here are just gold. I really invite you to read it. And then Schmidhuber, of course, being Schmidhuber, replies again. Down here, he has a response to the reply. And I don't expect Hinton to reply again, so I waited for a bit. But I believe Hinton when he says he does it only once. So he goes into this. It just says, summary, the facts presented in sections one, two, three, four, five are still valid. So he goes kind of statement by statement. Is this having a public debate? Blah, blah, blah. And he just says, well, this is an ad hominem attack, which is true. Right. This is true. And he says he even has multiple aliases in Wikipedia. And he just says another ad hominem attack. And then he goes into that Schmidhuber tries to discredit Alan Turing. And then Schmidhuber goes into this big, long, big, long, basically claim that Alan Turing wasn't as important as people made him out to be. And people invented this kind of Turing machine equivalence before that. Again, it's kind of Schmidhuber's take that the idea basically was already there and these people don't get the correct credit. And also he's correct that this is a true it's an ad hominem attack. Right. So, you know, be it as it may. This is correct. And then when Hinton goes that he didn't invent Backprop and Schmidhuber says this is finally a response related to my post, which is true. Right. However, it does not at all contradict what I wrote. And it is true that Hinton credited his co-author Rommelhardt with the invention, but neither cited Lin-Neymar and also the statement, lots of people. He says it wasn't created by lots of different people, but exactly one person. So this I find, like, can you really say now this is the exact time when Backprop was invented, even though it probably wasn't in the exact current formulation. And it probably existed somewhat like this. So, but again, and he his main claim is Dr. Hinton accepted the Honda price, although he apparently agrees that Honda's claims are false. He should ask Honda to correct their statements. And maybe you're going to would like to set the record straight. We invented LSTMs and, you know, as we as you may know, Sepp Hochreiter kind of invented LSTMs under Jürgen Schmidhuber as a PhD advisor. But the to summarize Dr. Hinton's comments and ad hominem arguments diverge from the contents of my post and do not challenge the facts and so on. And I have to say after reading this, this is this is correct. Right. Hinton basically replies to, hey, I, I never claimed I invented Backprop and other people have invented it. And Schmidhuber doesn't criticize Hinton in this particular post. He may otherwise. Schmidhuber doesn't criticize Hinton for claiming that he criticizes Honda for claiming that Hinton did. And Hinton doesn't. Hinton basically agrees with him. And also Schmidhuber says Dr. Hinton accepted the Honda price, although he apparently agrees that the claims are false. He should ask Honda to correct their statements. And it is true that Hinton accepted this price under this release. Right. Now, you might be able to say Hinton also says he's on the record basically saying he didn't do this. And I guess if you're Hinton and you know, you've had this you've had the successful career and so on. And you have previously really publicly stated that you didn't invent these things and, you know, made it clear. And then you get a prize and they write this thing. Maybe you just don't want to go after every single press statement and correcting that. But, you know, in essence, basically Hinton understood this as an attack on himself that he claims he invented Backprop. And Schmidhuber says Honda claims Hinton invented Backprop and Hinton accepted the price. So he agrees with it and Hinton basically agrees with it, but doesn't say Honda should have corrected it, which I can understand. So this is my take on this issue. It's kind of both are correct and they just kind of talk past each other. And Schmidhuber is always on the idea existed before. And Hinton is correct when he says it's not always just about the idea. Progress is also made by people being excited, people actually getting something to work, people, you know, doing something at the right time, the right place, which is also correct. But it is fun. It is fun. So I just I enjoyed I enjoy this honestly, like because ultimately this is the kind of discussions also need to happen in science because credit assignment is an important thing in science. And even though sometimes it's over the top, like Schmidhuber always going after it, I think we need people like him just kind of to keep the field in check a bit. And yeah, I will link to all of this. I hope you enjoy this and I wish you a nice rest of the weekend. If you're still here, consider subscribing and leave comment if you want. I usually read them. Bye bye.
 On April 21st, Jürgen Schmidhuber tweeted out,
 Stop crediting the wrong people for inventions made by others.
 At least in science, the facts will always win at the end,
 as long as the facts have not yet won.
 It is not yet the end.
 No fancy award can ever change that.
 Hashtag self-correcting science, hashtag plagiarism.
 And links to an article of his own website,
 where he wrote,
 Critique of Honda Prize for Dr. Hinton.
 So this is on Schmidhuber's own website,
 and it's by himself.
 Don't you love this?
 How to pronounce his name, Jürgen Schmidhuber.
 You again. Sorry.
 This is absolutely great.
 So both, actually, Schmidhuber and Hinton are on Twitter.
 You can tweet at them and follow them.
 This article here is basically a critique
 of the press release of Honda
 when they awarded Jeff Hinton
 for his achievements.
 And it goes through it step by step.
 And we won't look at the whole thing,
 but just for you to get the flavor.
 So here Honda says,
 Dr. Hinton has created a number of technologies
 that have enabled the broader application of AI,
 including the backpropagation algorithm
 that forms the basis of deep learning approach to AI.
 And Schmidhuber just goes off.
 He basically claims,
 while Hinton and his coworkers have made
 certain significant contributions to deep learning,
 the claim above is plain wrong.
 He says Hinton did not invent backpropagation.
 The person who invented backpropagation was Seppo Linayma.
 He says basically many papers failed to cite Linayma,
 who was the original inventor of backprop and so on.
 And he goes through a history of this
 and how it's even earlier.
 I always have a bit of a trouble with claims
 like who invented what,
 because when is an algorithm really the same thing?
 And when is it a variation on another algorithm?
 And when is it something completely new?
 It's never entirely clear.
 But the points here made that the things,
 the backpropagation algorithm existed before Hinton.
 And also that some of the papers,
 some of the seminal papers did not cite the correct origin.
 Statement 2. In 2002 he introduced
 a fast learning algorithm for restricted Boltzmann machines
 that allowed them to learn a single layer of distributed representation
 without requiring any labeled data.
 These methods allowed deep learning to work better
 and they led to the current deep learning revolution.
 And he basically goes, no, Dr. Hinton's interesting unsupervised pre-training
 for deep neural networks was irrelevant
 for the current deep learning revolution.
 In 2010 our team showed that the feed forward networks
 can be trained by plain backprop,
 do not at all require pre-training.
 And he basically again says,
 apart from this Hinton's unsupervised pre-training
 was conceptually a rehash of my unsupervised pre-training
 for deep recurrent neural networks.
 So he, you know, as you know, Schmidhuber has done a lot of work
 in recurrent neural networks and he basically says
 it was just a rehash of his algorithm.
 Now I have to say I have,
 so first of all, he makes a point here, right,
 that we don't really do unsupervised pre-training anymore
 until now, of course.
 But you, like, to train an MNIST classifier,
 you don't have to do that.
 But it's also doubtful that this was a step,
 even though if it wasn't on the exact path
 to the current situation,
 it was a thing that got people excited maybe.
 And so the critique is like half valid.
 And also it doesn't help Schmidhuber
 that he always compares it to his own things.
 Like it just, like, either criticize them for, you know,
 general things, but then avoid bringing your own things in
 because it just sounds like I did this before.
 And also I read some papers from these times.
 People just wrote papers sometimes.
 I haven't read this specific one,
 but sometimes people just wrote papers
 writing down their ideas.
 Like, one could do this and this and this.
 Never doing any experiments
 or actually specifying exactly what they mean.
 They just kind of wrote down a bunch of ideas
 and that got published.
 Especially, like, there's some reinforcement learning papers
 where people are just like, oh, one,
 I imagine agents doing this and learning from that.
 So it is, again, it is never really clear.
 Ideas are just had by everyone.
 I think people mistake this,
 that think that ideas are unique.
 It's not ideas that are unique.
 Many people have the same ideas, but some...
 There's also execution and exact formalization and so on.
 And exact level of specificity.
 All of this is really hard.
 And then the Honda says, in 2009, Dr. Hinton
 and two of his students used multilayer neural nets
 to make major breakthrough in speech recognition.
 That led directly to greatly improved.
 And this, of course, Schmidrub goes off by this
 because speech recognition is, of course, prime LSTM territory.
 So you don't want to go near this.
 And the Honda further says,
 revolutionized computer vision
 by showing that deep learning worked far better
 than existing state of the art.
 And again, he says the basic ingredients
 were already there and so on.
 And our team in Switzerland already used
 his first superior award-winning GPU-based CNN and so on.
 That's what was called DanNet, was produced by his group.
 And again, this seems correct, right?
 This seems when he lays it out like this,
 it doesn't change the fact that AlexNet won ImageNet in 2012.
 And that was like the start of the deep learning revolution.
 It was like, wow, you can cut the error rate by something like 30%
 simply by doing this deep learning stuff.
 So again, even if DanNet says it blew away the competition,
 it always seems like Schmidhuber is kind of right,
 but then also he's not.
 He's like, exact academic work
 and the idea being there on a paper
 isn't the only thing that drives progress.
 And it says, to achieve their dramatic results,
 Dr. Hinton also invented a widely used new method called dropout,
 which uses overfitting.
 No, like no, like no, just no.
 Randomly dropping parts in order to make something more robust,
 that is surely not a new thing.
 And he also says much earlier, there's this stochastic delta rule and so on.
 And he also critiques that this paper did not cite this.
 They just gave it the name.
 This is an idea that is kind of so simple
 that you wouldn't even necessarily think about researching
 whether that has existed already.
 I think they just did it because it's a natural idea,
 and then they gave it a name, and the name stuck.
 It's not about the idea itself.
 And then lastly, they say, of the countless AI-based technological services
 across the world, it is no exaggeration to say that few would have been possible
 without the results Dr. Hinton created.
 I love this.
 Name one that would not have been possible.
 And he just gives a list of their own group
 that are basically possible without Hinton's contributions.
 And this is just a bit of a cheap shot.
 Clearly, Honda, they're not saying it would have been physically impossible.
 Without his contributions.
 But certainly Hinton has, even if he hadn't invented any of those things,
 he certainly has created a spark.
 And these things created a splash, got people excited,
 people thinking about new ways of applying things, even if this is all true.
 But I would like you to notice,
 this is a critique of what Honda says about Hinton.
 And if I read through the statements of Schmidhuber,
 most of them are technically correct.
 And you know, that was that.
 And then I thought, OK, cool.
 But then someone posted it on Reddit.
 And then Hinton replies.
 And this is...
 Don't you love this?
 So Hinton says,
 Having a public debate with Schmidhuber about academic credit is not advisable
 because it just encourages him.
 And there is no limit to the time and effort that he is willing to put
 into trying to discredit his perceived arrivals.
 He is even escorted to tricks like having multiple aliases in Wikipedia
 to make it look as if other people agree.
 The page on his website about Alan Turing is a nice example
 of how he goes on trying to...
 These are shots fired.
 And he says, I'm going to respond once and only once.
 I have never claimed that I invented backpropagation.
 David Rumelhard invented it independently.
 After other people in other fields had invented it.
 It's true. When we first published, we did not know the history.
 So he basically says, OK, we did forget to cite it when we first published
 about backprop, but he doesn't say he invented it.
 What I've claimed is that I was the person to clearly demonstrate that
 backprop could learn interesting internal representations
 and that this is what made it popular.
 So this goes into the direction.
 Schmidhuber is very much on academic contributions.
 Idea was there before.
 And Hinton basically says, no, what we did is kind of we showed
 that it works in this particular way and we kind of got people excited about it.
 I did this by forcing that, blah, blah, blah.
 And he says, it is true that many people in the press have said
 I invented backprop and I've spent a lot of time correcting them.
 Here is an excerpt from 2018 where this is, I guess, a quote from this book
 that quotes Hinton where he says, lots of people invented different versions
 of backprop before David Rumelhart.
 They were mainly independent inventions, something I feel I've got too much credit for.
 It's one of these rare cases where an academic feels he has gotten too much credit for something.
 My main contribution was to show you can use it for learning distributor representations.
 So I'd like to set the record straight on that.
 Then he says, maybe Juergen would like to set the record straight on who invented LSTMs.
 Boom, boom.
 Crazy.
 Shots fired by Hinton here.
 This is, I mean, this is just great.
 But again, look at what Hinton says.
 Hinton basically says, yes, I have not invented that.
 I have corrected this on public record in the past.
 And yeah, so that's what Hinton says.
 And I mean, the comments here are just gold.
 I really invite you to read it.
 And then Schmidhuber, of course, being Schmidhuber, replies again.
 Down here, he has a response to the reply.
 And I don't expect Hinton to reply again, so I waited for a bit.
 But I believe Hinton when he says he does it only once.
 So he goes into this.
 It just says, summary, the facts presented in sections one, two, three, four, five are still valid.
 So he goes kind of statement by statement.
 Is this having a public debate? Blah, blah, blah.
 And he just says, well, this is an ad hominem attack, which is true.
 Right. This is true.
 And he says he even has multiple aliases in Wikipedia.
 And he just says another ad hominem attack.
 And then he goes into that Schmidhuber tries to discredit Alan Turing.
 And then Schmidhuber goes into this big, long, big, long, basically claim that Alan Turing wasn't as important as people made him out to be.
 And people invented this kind of Turing machine equivalence before that.
 Again, it's kind of Schmidhuber's take that the idea basically was already there and these people don't get the correct credit.
 And also he's correct that this is a true it's an ad hominem attack.
 Right. So, you know, be it as it may.
 This is correct.
 And then when Hinton goes that he didn't invent Backprop and Schmidhuber says this is finally a response related to my post, which is true.
 Right. However, it does not at all contradict what I wrote.
 And it is true that Hinton credited his co-author Rommelhardt with the invention, but neither cited Lin-Neymar and also the statement, lots of people.
 He says it wasn't created by lots of different people, but exactly one person.
 So this I find, like, can you really say now this is the exact time when Backprop was invented, even though it probably wasn't in the exact current formulation.
 And it probably existed somewhat like this.
 So, but again, and he his main claim is Dr. Hinton accepted the Honda price, although he apparently agrees that Honda's claims are false.
 He should ask Honda to correct their statements.
 And maybe you're going to would like to set the record straight.
 We invented LSTMs and, you know, as we as you may know, Sepp Hochreiter kind of invented LSTMs under Jürgen Schmidhuber as a PhD advisor.
 But the to summarize Dr. Hinton's comments and ad hominem arguments diverge from the contents of my post and do not challenge the facts and so on.
 And I have to say after reading this, this is this is correct.
 Right. Hinton basically replies to, hey, I, I never claimed I invented Backprop and other people have invented it.
 And Schmidhuber doesn't criticize Hinton in this particular post.
 He may otherwise. Schmidhuber doesn't criticize Hinton for claiming that he criticizes Honda for claiming that Hinton did.
 And Hinton doesn't. Hinton basically agrees with him.
 And also Schmidhuber says Dr. Hinton accepted the Honda price, although he apparently agrees that the claims are false.
 He should ask Honda to correct their statements.
 And it is true that Hinton accepted this price under this release.
 Right. Now, you might be able to say Hinton also says he's on the record basically saying he didn't do this.
 And I guess if you're Hinton and you know, you've had this you've had the successful career and so on.
 And you have previously really publicly stated that you didn't invent these things and, you know, made it clear.
 And then you get a prize and they write this thing.
 Maybe you just don't want to go after every single press statement and correcting that.
 But, you know, in essence, basically Hinton understood this as an attack on himself that he claims he invented Backprop.
 And Schmidhuber says Honda claims Hinton invented Backprop and Hinton accepted the price.
 So he agrees with it and Hinton basically agrees with it, but doesn't say Honda should have corrected it, which I can understand.
 So this is my take on this issue.
 It's kind of both are correct and they just kind of talk past each other.
 And Schmidhuber is always on the idea existed before.
 And Hinton is correct when he says it's not always just about the idea.
 Progress is also made by people being excited, people actually getting something to work, people, you know, doing something at the right time, the right place, which is also correct.
 But it is fun. It is fun.
 So I just I enjoyed I enjoy this honestly, like because ultimately this is the kind of discussions also need to happen in science because credit assignment is an important thing in science.
 And even though sometimes it's over the top, like Schmidhuber always going after it, I think we need people like him just kind of to keep the field in check a bit.
 And yeah, I will link to all of this.
 I hope you enjoy this and I wish you a nice rest of the weekend.
 If you're still here, consider subscribing and leave comment if you want.
 I usually read them. Bye bye.