Peer Review is still BROKEN! The NeurIPS 2021 Review Experiment (results are in)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper neurips neurips experiment peer review experiment neurips peer review peer review agreement neurips conference machine learning conference ai conference machine learning peer review peer review process peer review broken peer review accuracy reviewer number 2 neurips 2014
#neurips #peerreview #machinelearning

A look at the results of the 2021 NeurIPS peer review experiment.

https://arxiv.org/abs/2109.09774
https://www.reddit.com/r/MachineLearning/comments/qzjuvk/discussion_neurips_2021_finally_accepted/

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Do you know how hard it is to truly generate random numbers? I don't mean the random number generator on your phone or anything like this. That's just algorithm that crunches something, but it's deterministic. True random numbers are super difficult to generate. There is even a Wikipedia article about it. What you need to do is you need to measure some actual physical phenomenon like atmospheric noise or thermal noise or other things that we have no idea. They are so chaotic. We just can't predict them and thus their results are truly, truly random. Random.org even sells true random number generators for you. This is big topic humanity has searched far and wide for truly random processes. But now, ladies and gentlemen, we found it. The NeurIPS review process is a absolutely truly random phenomenon. So if you're not aware, a way, way time ago in NeurIPS, what was that? 2014, the organizers made a little experiment where they gave certain set of papers that was submitted to the conference, not only to one committee to review, but the two separate committees in order to track how the committees would agree or disagree. Now, the results right there were quite damning, to be honest. So not only did they not find any sort of correlation between what the reviewers scores they gave with any sort of future citations, and that's a paper that I've covered in a video where they look back seven years later at whether or not the reviewers could predict anything about these papers. Turns out they cannot. They also found that the reviewers mostly didn't really agree that much. So here were these experiments. Now of the 166 papers, most were rejected by both committees, which most papers to such a conference are rejected. So reject is sort of the default answer. But here, look at that. If committee one accepted and committee one accepted for 22 plus 21 papers, so for 33 papers, committee two only agreed on half of them. And likewise, when committee two accepted for the 43 papers, and this is 44 papers, so for the 44 papers that committee two accepted, committee one only agreed again in half of them. So this means that if you were to switch committees for the papers, only half of the accepted papers would be the same papers, half of them would be other papers that had actually been rejected by the other committee, which is kind of crazy. But this just shows you how noisy this process really is. Now it's 2021. And we've actually repeated this experiment. So here's a Reddit post by the user ygochang that has scraped from open review these scores and put together some statistics, such as this one here that shows the average rating of the papers versus how many of papers were in a particular bucket, and what ultimately happened to them. So we only have full data insight into the accepted papers and the rejected papers that have sort of voluntarily agreed to make their reviews public, which most papers that are rejected don't. Now the most interesting part here is this one. This is the repetition of the NURIPS experiment. You can see at the bottom, the total is almost 300 papers. And again, these are not all the papers part of the experiment. These are only the papers that were accepted because we don't know anything about the other ones. So the way this worked was the follows. Papers were given to two separate committees. These two committees reached a decision independently of each other. And then the maximum of the two decisions was taken as an acceptance criterion. So if either of the committees accepted the paper to be published, the paper was going to be published. So to understand this table, the leftmost column is the final decision, which is the max of decision one and decision two, not always, but we'll get to that. Then the second column is the decision of the first committee. And the third column is the decision of the second committee. Now these things are ordered, so it's not the same as in the last paper I've shown you. So since there's no clear ordering, we simply always put the larger decision on the left and the second large decision on the right. So the most interesting part of this is how many papers were accepted by one committee but rejected by another one. For that we have to add together all the rows where one of the decision is a reject. So 174 plus 16 plus 9 is I think 199 papers. 199 papers out of the 298 papers that were accepted had actually been rejected by a second committee. So to compare we have to do the following. We'll say that essentially the analogy would be that 22 and 22 and 21 papers, so 65 papers would be our analogous total number from down here. Those are the papers that ultimately ended up being accepted because they were accepted by one of the committees. And then 22 plus 21 papers, so 43 papers, would be the amount of papers that would have been rejected by one of the two committees but ultimately ended up being accepted because it was accepted by the other one. So according to this here we see 43 out of 65 papers only were accepted by one of the committees and here we see that roughly 200 out of 300 papers were only accepted by one of the committees. In both cases it's about two-thirds of the paper which means that actually this is remarkably consistent. So in the face of that and with the explosion of the machine learning community, more papers, more reviewers and so on, you could actually say it's a good thing. It's actually surprising this hasn't gotten much worse over the years. Now that's one way to look at it and the other way to look at it is to say this is crap. Like come on this is completely inconsistent. Not only the accept reject is inconsistent, you see of the six papers suggested to an oral by one of the committees, this was never confirmed by another committee. And how many were suggested for a spotlight by one of the committees? 16, 20, 29, 41, 44. 44 papers were suggested for a spotlight by one of the committees yet only three had actually both committees agreeing. And again the same results hold. If you were to swap out committees, if you just differently assign people to papers, half of the papers that are in the conference would be different. Half. I don't know how people can still claim that peer review is like this esteemed thing that is supposed to catch errors and do quality control and yada yada yada. There's something to be said that if you have a really good paper, the probability that a different committee also accepts it is pretty high. And also if you have a really bad paper, the probability that two committees agree on rejecting it, I guess that's even higher. However, most papers fall somewhere in the middle and that's the area of true randomness. Essentially what you do is you throw your paper in there and then something something happens and then you get a random number at the end. And remember people use this to justify archive blackouts, social media blackouts. Oh my god, you cannot bias the reviewers. You must not bias the pristine review. Like how? You cannot bias a random number generator. I guess you can but it makes no sense. Like honestly, this is only half joking at this point. The social media networks that we have, people surfacing interesting papers from the depths of archive and from their social networks, all the people filtering this kind of stuff. Yes, there's promotion going on. Yes, there's hype. Yes, money plays a role. But still, this is a much better process than just like three random dudes sitting on the toilet like scrolling through your paper a bit and then writing, not enough experiments. Reject. I don't understand it. It's confusing. Look at the learning rate grafting video I did. Like these are the types of reviews that reviewers have to battle with. Yes, it hasn't gotten much worse over the years. Yes, really good papers are consistent, really bad papers are consistent. But I still maintain that this situation is not really a good one. This is absolutely inconsistent. It's a lottery. Your best bet is to write as many papers as you can that are just barely, barely not crap and then throw all of them in and through the random number process, some of them will get accepted. And that's a sad state because big companies do this for clout. Big companies do it to recruit new people and so on. But there are a lot of PhD students that need to get whatever their three papers published in their four or five years that they're doing the PhD. And with such randomness, and with only very, very limited amount of conferences that you can submit to over the course of a year, there's like three or four different big conferences that you realistically can submit to if you want a good impact factor. This is very bad situation. And a lot of people are going to be damaged just because the universe has some random fluctuations. The solution to this honestly, starts with professors, tenured professors start handing out PhDs independent of conference submissions, universities start giving professors tenure, not on the basis of the impact factor of where they publish, look at citations, look at how popular the work is in any other metric, stop considering impact factors of conferences, grant agencies, stop giving out grants based on the reputations of the professors based on the impact factors, essentially disregard conference publications for anything you do. I see some people, they have to do it. Some professors have to get tenure. And this is a criteria and PhD students have to do this because that's a requirement for their PhD. But if you're in a position to discard all of this, do it. What stops you you have tenure tell your PhD students do three really nice really good archive publications if I'm happy with it PhD. Alright, that was it from me for ranting about this topic. What do you think about it? Let me know in the comments. Maybe I'm completely wrong here. But you know, I'm happy to be educated to the contrary. See ya.
 Do you know how hard it is to truly generate random numbers? I don't mean the random number
 generator on your phone or anything like this. That's just algorithm that crunches something,
 but it's deterministic. True random numbers are super difficult to generate. There is even a
 Wikipedia article about it. What you need to do is you need to measure some actual physical
 phenomenon like atmospheric noise or thermal noise or other things that we have no idea. They are so
 chaotic. We just can't predict them and thus their results are truly, truly random. Random.org even
 sells true random number generators for you. This is big topic humanity has searched far and wide
 for truly random processes. But now, ladies and gentlemen, we found it. The NeurIPS review process
 is a absolutely truly random phenomenon. So if you're not aware, a way, way time ago in NeurIPS,
 what was that? 2014, the organizers made a little experiment where they gave certain set of papers
 that was submitted to the conference, not only to one committee to review, but the two separate
 committees in order to track how the committees would agree or disagree. Now, the results right
 there were quite damning, to be honest. So not only did they not find any sort of correlation between
 what the reviewers scores they gave with any sort of future citations, and that's a paper that I've
 covered in a video where they look back seven years later at whether or not the reviewers could
 predict anything about these papers. Turns out they cannot. They also found that the reviewers
 mostly didn't really agree that much. So here were these experiments. Now of the 166 papers,
 most were rejected by both committees, which most papers to such a conference are rejected. So
 reject is sort of the default answer. But here, look at that. If committee one accepted and
 committee one accepted for 22 plus 21 papers, so for 33 papers, committee two only agreed
 on half of them. And likewise, when committee two accepted for the 43 papers, and this is 44 papers,
 so for the 44 papers that committee two accepted, committee one only agreed again in half of them.
 So this means that if you were to switch committees for the papers, only half of the accepted papers
 would be the same papers, half of them would be other papers that had actually been rejected by
 the other committee, which is kind of crazy. But this just shows you how noisy this process really
 is. Now it's 2021. And we've actually repeated this experiment. So here's a Reddit post by the
 user ygochang that has scraped from open review these scores and put together some statistics,
 such as this one here that shows the average rating of the papers versus how many of papers
 were in a particular bucket, and what ultimately happened to them. So we only have full data
 insight into the accepted papers and the rejected papers that have sort of voluntarily agreed to
 make their reviews public, which most papers that are rejected don't. Now the most interesting part
 here is this one. This is the repetition of the NURIPS experiment. You can see at the bottom,
 the total is almost 300 papers. And again, these are not all the papers part of the experiment.
 These are only the papers that were accepted because we don't know anything about the other ones.
 So the way this worked was the follows. Papers were given to two separate committees. These two
 committees reached a decision independently of each other. And then the maximum of the two decisions
 was taken as an acceptance criterion. So if either of the committees accepted the paper to be
 published, the paper was going to be published. So to understand this table, the leftmost column
 is the final decision, which is the max of decision one and decision two, not always,
 but we'll get to that. Then the second column is the decision of the first committee. And the
 third column is the decision of the second committee. Now these things are ordered, so it's
 not the same as in the last paper I've shown you. So since there's no clear ordering, we simply always
 put the larger decision on the left and the second large decision on the right. So the most
 interesting part of this is how many papers were accepted by one committee but rejected by another
 one. For that we have to add together all the rows where one of the decision is a reject. So 174 plus
 16 plus 9 is I think 199 papers. 199 papers out of the 298 papers that were accepted had actually
 been rejected by a second committee. So to compare we have to do the following. We'll say that
 essentially the analogy would be that 22 and 22 and 21 papers, so 65 papers would be our analogous
 total number from down here. Those are the papers that ultimately ended up being accepted because
 they were accepted by one of the committees. And then 22 plus 21 papers, so 43 papers, would be the
 amount of papers that would have been rejected by one of the two committees but ultimately ended up
 being accepted because it was accepted by the other one. So according to this here we see 43 out
 of 65 papers only were accepted by one of the committees and here we see that roughly 200 out
 of 300 papers were only accepted by one of the committees. In both cases it's about two-thirds
 of the paper which means that actually this is remarkably consistent. So in the face of that and
 with the explosion of the machine learning community, more papers, more reviewers and so on,
 you could actually say it's a good thing. It's actually surprising this hasn't gotten much worse
 over the years. Now that's one way to look at it and the other way to look at it is to say this is
 crap. Like come on this is completely inconsistent. Not only the accept reject is inconsistent, you see
 of the six papers suggested to an oral by one of the committees, this was never confirmed by
 another committee. And how many were suggested for a spotlight by one of the committees? 16, 20, 29,
 41, 44. 44 papers were suggested for a spotlight by one of the committees yet only three had actually
 both committees agreeing. And again the same results hold. If you were to swap out committees,
 if you just differently assign people to papers, half of the papers that are in the conference would
 be different. Half. I don't know how people can still claim that peer review is like this esteemed
 thing that is supposed to catch errors and do quality control and yada yada yada. There's
 something to be said that if you have a really good paper, the probability that a different
 committee also accepts it is pretty high. And also if you have a really bad paper, the probability
 that two committees agree on rejecting it, I guess that's even higher. However, most papers fall
 somewhere in the middle and that's the area of true randomness. Essentially what you do is you
 throw your paper in there and then something something happens and then you get a random
 number at the end. And remember people use this to justify archive blackouts, social media blackouts.
 Oh my god, you cannot bias the reviewers. You must not bias the pristine review. Like how?
 You cannot bias a random number generator. I guess you can but it makes no sense. Like honestly,
 this is only half joking at this point. The social media networks that we have, people surfacing
 interesting papers from the depths of archive and from their social networks, all the people
 filtering this kind of stuff. Yes, there's promotion going on. Yes, there's hype. Yes, money
 plays a role. But still, this is a much better process than just like three random dudes sitting
 on the toilet like scrolling through your paper a bit and then writing, not enough experiments. Reject.
 I don't understand it. It's confusing. Look at the learning rate grafting video I did. Like these are
 the types of reviews that reviewers have to battle with. Yes, it hasn't gotten much worse over the
 years. Yes, really good papers are consistent, really bad papers are consistent. But I still
 maintain that this situation is not really a good one. This is absolutely inconsistent. It's a
 lottery. Your best bet is to write as many papers as you can that are just barely, barely not crap
 and then throw all of them in and through the random number process, some of them will get accepted.
 And that's a sad state because big companies do this for clout. Big companies do it to recruit
 new people and so on. But there are a lot of PhD students that need to get whatever their three
 papers published in their four or five years that they're doing the PhD. And with such randomness,
 and with only very, very limited amount of conferences that you can submit to over the
 course of a year, there's like three or four different big conferences that you realistically
 can submit to if you want a good impact factor. This is very bad situation. And a lot of people
 are going to be damaged just because the universe has some random fluctuations. The solution to this
 honestly, starts with professors, tenured professors start handing out PhDs independent
 of conference submissions, universities start giving professors tenure, not on the basis of
 the impact factor of where they publish, look at citations, look at how popular the work is in any
 other metric, stop considering impact factors of conferences, grant agencies, stop giving out grants
 based on the reputations of the professors based on the impact factors, essentially disregard
 conference publications for anything you do. I see some people, they have to do it. Some professors
 have to get tenure. And this is a criteria and PhD students have to do this because that's a
 requirement for their PhD. But if you're in a position to discard all of this, do it. What
 stops you you have tenure tell your PhD students do three really nice really good archive publications
 if I'm happy with it PhD. Alright, that was it from me for ranting about this topic. What do you
 think about it? Let me know in the comments. Maybe I'm completely wrong here. But you know,
 I'm happy to be educated to the contrary. See ya.