NeurIPS 2020 Changes to Paper Submission Process
machine learning deep learning phd papers neurips nips conference submission society ethics
My thoughts on the changes to the paper submission process for NeurIPS 2020.

The main new changes are:
1. ACs can desk reject papers
2. All authors have to be able to review if asked
3. Resubmissions from other conferences must be marked and a summary of changes since the last submission must be provided
4. Borader societal / ethical impact must be discussed
5. Upon acceptance, all papers must link to an explanatory video and the PDFs for slides and poster

https://neurips.cc/Conferences/2020/CallForPapers
https://youtu.be/361h6lHZGDg

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there. So I just wanted to give a few quick thoughts about the changes to the NeurIPS submission process this year as opposed to last year. They've announced this on the website, on Twitter, with the video and so on, and I thought I might share some thoughts on that. And maybe some of you haven't heard yet in case you're planning to submit or thinking about it. So desk rejections. ACs, area chairs, have the ability to desk reject papers that they feel strongly are not going to be passable to the reviewers. They did an experiment last year where the ACs were simply supposed to mark submissions that they would desk reject, and it turned out that ACs aren't very good at estimating which submissions are going to be rejected by the reviewers. That might be because there wasn't really anything at stake because it was just kind of a let's see how this works. But it is definitely a move to reduce the number of submissions because the field is exploding and we lack reviewing power, reviewing people. So this is a move to reduce the number of people that have to review something because there will be fewer papers. I don't know if this increases the quality overall. If your paper gets desk rejected, there's usually some obvious reason for it why an AC decided it's not worth it. They probably haven't read it in depth, but there might be some kind of overall structural issue that, or like the introduction has many typos, or you know, look for the obvious things even though your work might be good. Second, all authors of a paper have to be able to review if asked to do so. And again, this is a stab at this kind of reviewing crisis, but I have mixed feelings about this. I really think this is a move in the wrong direction. It will increase the number of authors because a lot of people have been kind of free riding in that they're submitting papers, but they aren't reviewing other papers even though they would be competent researchers simply because reviewing doesn't get you anything. So there's no incentive to do reviews. Maybe you can say you're a reviewer, but then there's every incentive to do bad reviews, like two line reviews where the first line says you should have compared to my paper, reject like, fuck you if you're a reviewer like this. In any case, like a lot of times, and this hits, for example, like universities where you maybe work with a master student and the master student does some of the pre-processing of the data and they don't really have a clue about the machine learning, but they still contribute it. So why shouldn't they be an author on the paper? They might even have written that section about the data pre-processing. And now they're asked to review entire papers about topics where they're not really familiar with or you have some outside collaborators or, you know, there are so many things wrong. I think this attracts the wrong kind of people and by forcing people to do it, you encourage even more, like all these reviewers that would not have reviewed, what will happen is they will give shitty reviews and you will have even worse quality of reviews as a result. I think this is the wrong move to reduce the number of load per reviewer. I'd rather see abolish peer review completely in computer science, in machine learning at least. That's my opinion, but that might be a video for another time. I have plans how to replace it another time. Resubmissions have to be clearly marked. So if your paper is a resubmission of, like if you had already submitted it in the last 12 months, it's been rejected, you have to say it is a resubmission and the changes you made to the paper. Again with a peer review process that actually works, this would make a lot of sense. You can say, well, it got rejected last time and here is how I corrected for what the reviewers criticized, but with the review quality right now, I mean most of the papers, what are they going to say? It got rejected for nefarious reasons because the reviewer had a bad bowel movement that morning and I didn't really change much. So you encourage people to kind of blow out of proportion the changes they made and put a lot of additional unnecessary work on two papers that would actually be already fine. So all of these things, they are forcing people to do things and then the incentives of what we want aren't aligned with what we give. So what you'll end up with is lower quality reviews and lower quality work. So the next two points are of a different nature. The first one though, that will probably, I mean even if the ACs aren't perfect, that's a good move. I like that. The fourth point and the fifth point are a bit different. The fourth point is there is a new section in CMT apparently where you have to describe the broader societal impact and ethics around your work. How will your work influence society? What are positives and negatives? Ethical outcomes? How can it be used? And this is targeted towards things like let's say facial recognition. If you develop a new facial recognition algorithm, you may be able to argue, well this could be better used to identify victims in a big crowd. There's a mass riot or something and then you don't know who is there. Is my relative one of the people in the mass that gets stomped on? Or you can also say this potentially helps a dictatorial state to govern their people because they can now recognize everyone. For most papers it will be a bit shaky. Like if your third order optimization algorithm achieves a slightly better convergence rate, I'm not sure what's here. But what I feel is that this is dumb in a way because this just means more work. Basically now you have to demonstrate and yeah it says you should discuss positive and negative aspects but in essence everyone will be demonstrating virtue signaling how good their work will be for society and what good can be done and maybe a bit of bad. But that can be mitigated and it just pushes into a more PR world. So it goes from the science world into a more PR world. It means extra work and who are the people that can afford to do extra work? It's mostly the big companies. They can just put an additional team member on that, maybe even do additional experiments to show the societal impact of the work and who will lose out are probably small universities, independent researchers. And so on that don't have that capacity that simply do their research because it's an interesting research question. And for almost every single thing in the world that has an application it will have good and bad applications. So yeah mixed feelings. So the fifth is you are now supposed if your paper gets accepted to make a video about it and upload the poster basically link to the poster that you would use and also link to slides that you would give your talk with. This is to make it more accessible to people that are not at the conference which again I have mixed feelings about. Again it pushes it into this more PR realm. Talks are already live streamed. Most of them are for most of the large conferences and I feel it just gets people one step more away from the actual paper. So it allows people to grandstand and PR up even more of their work because even people who don't attend the conference now they're not going to read the paper, they're just going to watch the video. And in the video you can always leave away those things that you would have to like that a reviewer makes you put in the paper right and in the video you can overbought. It's camera ready. No one reviews the video. You can say whatever you want. So it's just where before if you didn't attend the conference I think many people actually did read the paper, watched talks where people could ask questions and now it's just one more PR thing. And again who has time, energy and money to really invest a lot into this? It's mainly large companies right if you're small and you're time bound and so on you might not have equipment or time to do that. I am not for hire to do your NURBS videos just saying. I don't have time to make these videos really. As you can see stellar quality I think there's a bright glare right here. So that was it for my opinions on this and I wish you a nice day. Bye bye.
 Hi there.
 So I just wanted to give a few quick thoughts about the changes to the NeurIPS submission
 process this year as opposed to last year.
 They've announced this on the website, on Twitter, with the video and so on, and I thought
 I might share some thoughts on that.
 And maybe some of you haven't heard yet in case you're planning to submit or thinking
 about it.
 So desk rejections.
 ACs, area chairs, have the ability to desk reject papers that they feel strongly are
 not going to be passable to the reviewers.
 They did an experiment last year where the ACs were simply supposed to mark submissions
 that they would desk reject, and it turned out that ACs aren't very good at estimating
 which submissions are going to be rejected by the reviewers.
 That might be because there wasn't really anything at stake because it was just kind
 of a let's see how this works.
 But it is definitely a move to reduce the number of submissions because the field is
 exploding and we lack reviewing power, reviewing people.
 So this is a move to reduce the number of people that have to review something because
 there will be fewer papers.
 I don't know if this increases the quality overall.
 If your paper gets desk rejected, there's usually some obvious reason for it why an
 AC decided it's not worth it.
 They probably haven't read it in depth, but there might be some kind of overall structural
 issue that, or like the introduction has many typos, or you know, look for the obvious things
 even though your work might be good.
 Second, all authors of a paper have to be able to review if asked to do so.
 And again, this is a stab at this kind of reviewing crisis, but I have mixed feelings
 about this.
 I really think this is a move in the wrong direction.
 It will increase the number of authors because a lot of people have been kind of free riding
 in that they're submitting papers, but they aren't reviewing other papers even though
 they would be competent researchers simply because reviewing doesn't get you anything.
 So there's no incentive to do reviews.
 Maybe you can say you're a reviewer, but then there's every incentive to do bad reviews,
 like two line reviews where the first line says you should have compared to my paper,
 reject like, fuck you if you're a reviewer like this.
 In any case, like a lot of times, and this hits, for example, like universities where
 you maybe work with a master student and the master student does some of the pre-processing
 of the data and they don't really have a clue about the machine learning, but they still
 contribute it.
 So why shouldn't they be an author on the paper?
 They might even have written that section about the data pre-processing.
 And now they're asked to review entire papers about topics where they're not really familiar
 with or you have some outside collaborators or, you know, there are so many things wrong.
 I think this attracts the wrong kind of people and by forcing people to do it, you encourage
 even more, like all these reviewers that would not have reviewed, what will happen is they
 will give shitty reviews and you will have even worse quality of reviews as a result.
 I think this is the wrong move to reduce the number of load per reviewer.
 I'd rather see abolish peer review completely in computer science, in machine learning at
 least.
 That's my opinion, but that might be a video for another time.
 I have plans how to replace it another time.
 Resubmissions have to be clearly marked.
 So if your paper is a resubmission of, like if you had already submitted it in the last
 12 months, it's been rejected, you have to say it is a resubmission and the changes
 you made to the paper.
 Again with a peer review process that actually works, this would make a lot of sense.
 You can say, well, it got rejected last time and here is how I corrected for what the reviewers
 criticized, but with the review quality right now, I mean most of the papers, what are they
 going to say?
 It got rejected for nefarious reasons because the reviewer had a bad bowel movement that
 morning and I didn't really change much.
 So you encourage people to kind of blow out of proportion the changes they made and put
 a lot of additional unnecessary work on two papers that would actually be already fine.
 So all of these things, they are forcing people to do things and then the incentives of what
 we want aren't aligned with what we give.
 So what you'll end up with is lower quality reviews and lower quality work.
 So the next two points are of a different nature.
 The first one though, that will probably, I mean even if the ACs aren't perfect, that's
 a good move.
 I like that.
 The fourth point and the fifth point are a bit different.
 The fourth point is there is a new section in CMT apparently where you have to describe
 the broader societal impact and ethics around your work.
 How will your work influence society?
 What are positives and negatives?
 Ethical outcomes?
 How can it be used?
 And this is targeted towards things like let's say facial recognition.
 If you develop a new facial recognition algorithm, you may be able to argue, well this could
 be better used to identify victims in a big crowd.
 There's a mass riot or something and then you don't know who is there.
 Is my relative one of the people in the mass that gets stomped on?
 Or you can also say this potentially helps a dictatorial state to govern their people
 because they can now recognize everyone.
 For most papers it will be a bit shaky.
 Like if your third order optimization algorithm achieves a slightly better convergence rate,
 I'm not sure what's here.
 But what I feel is that this is dumb in a way because this just means more work.
 Basically now you have to demonstrate and yeah it says you should discuss positive and
 negative aspects but in essence everyone will be demonstrating virtue signaling how good
 their work will be for society and what good can be done and maybe a bit of bad.
 But that can be mitigated and it just pushes into a more PR world.
 So it goes from the science world into a more PR world.
 It means extra work and who are the people that can afford to do extra work?
 It's mostly the big companies.
 They can just put an additional team member on that, maybe even do additional experiments
 to show the societal impact of the work and who will lose out are probably small universities,
 independent researchers.
 And so on that don't have that capacity that simply do their research because it's an interesting
 research question.
 And for almost every single thing in the world that has an application it will have good
 and bad applications.
 So yeah mixed feelings.
 So the fifth is you are now supposed if your paper gets accepted to make a video about
 it and upload the poster basically link to the poster that you would use and also link
 to slides that you would give your talk with.
 This is to make it more accessible to people that are not at the conference which again
 I have mixed feelings about.
 Again it pushes it into this more PR realm.
 Talks are already live streamed.
 Most of them are for most of the large conferences and I feel it just gets people one step more
 away from the actual paper.
 So it allows people to grandstand and PR up even more of their work because even people
 who don't attend the conference now they're not going to read the paper, they're just
 going to watch the video.
 And in the video you can always leave away those things that you would have to like that
 a reviewer makes you put in the paper right and in the video you can overbought.
 It's camera ready.
 No one reviews the video.
 You can say whatever you want.
 So it's just where before if you didn't attend the conference I think many people actually
 did read the paper, watched talks where people could ask questions and now it's just one
 more PR thing.
 And again who has time, energy and money to really invest a lot into this?
 It's mainly large companies right if you're small and you're time bound and so on you
 might not have equipment or time to do that.
 I am not for hire to do your NURBS videos just saying.
 I don't have time to make these videos really.
 As you can see stellar quality I think there's a bright glare right here.
 So that was it for my opinions on this and I wish you a nice day.
 Bye bye.