[Rant] REVIEWER #2: How Peer Review is FAILING in Machine Learning
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper research ml conference nips neurips icml iclr review peer review publishing accept reject citations conflict reviewer rebuttal area chair money free experiments theory crisis boom overloaded incentives incentive revise quality
#ai #research #peerreview

Machine Learning research is in dire straits as more people flood into the field and competent reviewers are scarce and overloaded. This video takes a look at the incentive structures behind the current system and describes how they create a negative feedback loop. In the end, I'll go through some proposed solutions and add my own thoughts.

OUTLINE:
0:00 - Intro
1:05 - The ML Boom
3:10 - Author Incentives
7:00 - Conference Incentives
8:00 - Reviewer Incentives
13:10 - Proposed Solutions
17:20 - A Better Solution
23:50 - The Road Ahead

PS: If it is not entirely clear to anyone already, stealing ideas as a reviewer is against most conferences' code of ethics and I disapprove of any such behavior. I mention it because it is being done regularly and good luck proving it in any particular case.

Sources:
https://thecognitivevortex.wordpress.com/category/phd/
https://susannapaasonen.org/2019/05/31/observations-on-peer-reviewing/
https://www.radicalhistoryreview.org/abusablepast/forum-1-1-on-peer-review/
https://www.meme-arsenal.com/en/create/meme/2012988
https://imgflip.com/i/1pydon
https://uqkdhanj.wordpress.com/2015/02/18/10-best-reviewer-comments-in-meme-part-2/
https://susannapaasonen.org/2019/05/31/observations-on-peer-reviewing/
https://www.memecreator.org/meme/what-if-i-told-you-reviewer-2-wanted-more-experiments/
https://www.emaze.com/@ATFTTRRF
https://thegradient.pub/neurips-2019-too-big/
https://www.videezy.com/backgrounds/6199-switzerland-flag-4k-motion-loop-stock-video
http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
https://twitter.com/tdietterich/status/1292217162103316481
https://www.pinterest.de/pin/192951165261323337/

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 It's review time, review time. So NeurIPS has recently released the reviews for submitted papers and pretty much everyone is not happy and I think the reason is that even though you have the reasonable reviewers of these conferences there is always always reviewer number two and reviewer number two leaves very short review says that either there are not enough experiments or the theory is too weak or the assumptions aren't warranted or they just don't like your face and that's why they give you a weak reject. Actually some of them think your paper is fantastic and give you a weak reject. So a lot of people are angry, upset, dissatisfied with the quality of the reviews in machine learning conferences and today I want to go look a bit into how this works, why this is the way that it is and what we could potentially do about it. So what's happening with publishing in ML? The system seems to be overloaded. There's so much attention in machine learning right now that there hasn't been a few years ago that there's a huge influx of new people wanting to publish in this field. That creates a lot of submissions and not enough reviewers to peer review these submissions. So a lot of reviewers are recruited that probably shouldn't be reviewers. I hear stories of undergrads being recruited as reviewers, people from way outside the fields, people that don't have time. So too many submissions, too few inexperienced and not really expert reviewers creates pretty much a random process and this was also shown in a few years ago in the then NIPS experiment where it showed that for most papers being accepted is pretty much a coin flip with a weighted coin. The natural response as an author is going to be you're going to submit even more papers. If it's a coin flip you can just submit whatever and there's a chance it might get in. Which of course only makes the problem worse. So this entire process of science where you submit your manuscript and then you get the reviews and then you try to improve it. It's completely broken because not only do you not care about the reviews the next set of reviewers at the next conference are going to be different. So no matter what you improve right now the next set of people will have completely different criticism. It just doesn't work like it is intended to work. The review process is basically just some kind of a random nuisance to people that they have to get through and at the same time people who are reviewers have every incentive to make it as hard as possible for the people that are submitting. So in order to analyze this I want to look at the incentives of the different groups in this process and kind of show how the incentive structure upholds this system that benefits pretty much everyone participating in it but creates a worse outcome for all of us. So first of all let's look at paper authors. What are your incentive if you're an author of a paper? First of all authors they want to get as many papers as possible as fast as possible. Now in the current conference system the fastness isn't really up for debate it's as fast as it is. However authors can simply upload their paper to archive and be as fast as they want there. Another incentive for authors is to have as little comments on your paper as possible because comments usually mean criticism and you don't want comments and especially you don't want public permanent comments. The good thing for authors right now is on archive comments aren't possible and conference reviews even if they're made public no one goes to look at them everyone just goes to archive. So authors right now are getting a pretty good deal with respect to not getting their work criticized. Authors are also incentivized to give as little credit to people as possible and again the current system is totally in favor of that. The no commenting on archive basically means that you can claim whatever you want and if someone wants to refute you they have to make a big deal out of it and basically write their own paper and again people will probably not find that. On the other hand in the conferences reviewers are supposed to detect when you're not giving proper credit to other people. However most reviewers don't do that. Going out and really looking if everything is credited properly is one of the most time-consuming tasks when you review a paper and most reviewers simply aren't going through that trouble. The only downside for most authors is even though all of this is pretty much in their favor a lot of them still require that stamp of approval that peer review accepted at a good conference. So their incentive is to keep submitting to conferences as many papers as possible. Basically count on that random process to get them accepted and after that they're just fine. They have the stamp of approval there's absolutely no requirement to revise it. There's absolutely no requirement to have other people comment further on the work. So I guess the complaining here right now is just about the noisy process and everyone complains that their particular paper which is at the behest of the noisy process as everyone else's paper got an unfair treatment in that random process which half the papers do probably more. The incentives in the systems are actually even bigger for what I call the big names. These are the big research institutions of companies or big name professors anyone that has some sort of reputation. People argue that anonymous reviewing is actually good for small authors good for unknown authors because it hides their identity and the big names basically aren't able to play their big name credit to a paper. However there's an easy way to know that this isn't the case. The big names are doing just fine. Here's the issue if you want your name to be attached to something you're gonna find a way to do it. People are suggesting archive blackout periods and whatnot anonymous submissions to archive. You have to realize that if someone wants to give some information to the public they are going to. In fact right now the big names are finding every possible way to have their names attached to things and massively increase their chances of getting through the anonymous peer review process. You got to realize if you're well connected not only do you have an advertising platform but you can also pretty easily find out who your area chairs are, who's reviewing, in which track your paper gets and so on. So allow me to be a little bit skeptical about the claim that we need more anonymity in this process. I think we need less. Second what are the incentives of the conferences itself? So the conference organizers they want to have a good reputation which basically means they want to be like a cool nightclub. Lots of people want to get in but they have to reject a lot of those people in order to make the club exclusive and have a high reputation. So conferences have every reason to invite everyone to submit as much as possible but then to reject as much as possible to make it seem like it's super hard to get in. This only makes the problem worse and I think the current explosion isn't really desired by the conferences. As the process is super noisy they're slowly losing their reputation that way but still the incentives aren't to lower the amount of submissions and increase the overall quality because that means a higher percentage of submissions will have to get accepted which means that the conference appears to be less exclusive. And lastly let's look at the reviewers themselves. This is the most screwed up part in the system. I have every incentive to be a reviewer for one of these conferences because I can write that on my CV. Hey I was a reviewer at big name conference and then once I am accepted as a reviewer I have every incentive to do absolutely nothing. In fact the less time I waste with this the better because I'm not getting any public credit. I'm anonymous right? Anonymous peer review. I'm not getting any reputation out of this and in fact I can only lose from accepting papers and I can only lose from writing detailed reviews. If I'm short and vague and I reject a paper not only can I not really be criticized because I'm not saying much it's actually in my overwhelming interest if the paper has some sort of big mistake and I overlook it and I accept the paper and the other reviewers see that mistake this looks really really bad for me even though I'm anonymous in the broader context it also looks bad for the area chair supervising me if they don't see it it looks bad for the conference if their area chairs don't see it so there's a massive push to not make mistakes however if I reject a paper that was actually good I can just say well they can resubmit to the next conference. So I already have a giant prior to reject a paper add to that that usually the papers that I review might be my competition and by the conference incentive of being pretty exclusive the more of my competition gets accepted the less I might get accepted because not only are there limited amount of space not formally but informally other work might overlap substantially with my own and therefore make it less likely that I get published also other work might actually criticize my work and I don't like that and this is a bit cynical but I'm not saying everyone does this but there is an incentive for you as a reviewer especially if the work is close to what you're doing to reject it now implement the same or a very similar idea and then submit to the next conference where these other authors also will submit and hope for the random process to just for your paper to get more lucky than their paper. Black planting on archive counters this a little bit but I'm afraid that with proposed solutions like more archive blackout periods more anonymity these problems will only get worse maybe some people don't realize this but as a reviewer it's really easy for me to reject a paper I can almost always find reasons to reject a paper if it's a theory paper I can ask for experiments if it's an experimental paper I can ask for more experiments why didn't you test that data set why didn't you compare against this method why are your assumptions so strong they're never guaranteed in practice is the problem even relevant your theory is too weak have you looked at this other special case and if I really want to I can just ask many many many questions not even criticisms just many questions and I know the authors just have a one-page rebuttal they can never answer all my questions if I do that and then I can simply argue the authors failed to address all my questions properly so you might be asking why do some reviewers actually do a good job and that is I believe really a lot to do with good will most people are actually well intended most people actually want to do a good job in reviewing have the ethos of science and do take the time to do the reviews even though they're incentivized to do them badly even though they're incentivized to reject papers a lot of people still do a good job however reviewer number two usually doesn't and it only needs a very few reviewer number twos to make the field a whole lot worse now there's a question to be said aren't we all a bit reviewer number two have you ever written a review that the authors might think is completely unreasonable and while there is some truth to that argument I definitely know that there are differences in reviews in fact I've heard people brag about writing two line reviews where the second line is you didn't cite and compare to my own work and then laugh about that so goodwill won't carry us all the way if the incentive structure is bad and I believe most of this is because we've taken out the reputation game out of the review process in smaller fields of science it used to be that the journal editors knew the reviewers and their reputation at least towards the journal editor was on the line for all of future if they did a bad job right now everything's so big so anonymous people hardly remember the names of their co-reviewers no reputation is being damaged by bad reviews and that's how we get here so of course I'm not the first one to observe these problems so many people have proposed solutions and most of these solutions fall into the basis of what I would call a C based methods which is basically where someone evaluates the reviews while the reviews remain anonymous and that someone is usually the area chair so right now the area chair can already decide that a reviewer is really bad and then the reviewer will not be invited to review the next time around I just want to point out the irony of the situation conferences nowadays have so little reviewers that they require every author to be a reviewer but then your punishment for writing bad reviews is that you won't be invited to be a reviewer the next time around I mean can you make a better point that the system is failing of course the problem with all a C based methods is that you're basically moving a problem that has everything to do with people being unaccountable noisy not expertly having no time and every single incentive to do as little as possible you transfer that problem to even less people that have even less time that have even more stress that have an even broader view and topic area and are single people instead of three or four people so it's even more noisy if anything like this is implemented you'll just instead of seeing complaints about bad reviews in addition you will also see complaints about bad a C's that will certainly not make the problem any less in fact I would argue any a C based solution will make the problem worse other solutions are what I called payment based solutions like give the reviewers money to review I don't see how that fixes the incentive for you to reject anything you just might write it in a bit more eloquent style also as soon as you bring money into the game that automatically excludes a lot of people depending on how you do it that aren't as affluent which is certainly something we don't want as a community other people are pointing to things like open review which I agree is a better system however it is still anonymous so the same incentives exist and it is still a conference where you get a stamp of accept or reject and once it's accepted no one cares about the reviews anymore in fact in open review you can write as much text as you want so the a C's are even more overloaded with lots of text to make their decisions so something I want to highlight is a thread by Thomas G Dietrich on Twitter where he basically suggests some sort of a wiki and sort of a collaborative research wiki where you'd have a set of senior authors that basically maintain that wiki that do a first check of papers and kind of match them against the wiki of what's already known I won't go through that here I will link it and I definitely advise you to read it because it's very interesting proposal it's a sort of utopian dream I would actually welcome if we all work together on increasing the knowledge of mankind in a wiki style way however I think lots of people want their names attached to things and even if you do what Thomas suggests and basically have people write papers and then the editors integrate that into the wiki it is not clear how that system where the editors clearly need to be senior and experienced could deal any better with the explosion of research that we're dealing right now they would be as overloaded as the current system plus who's gonna be an editor Thomas says becoming an editor would be a very esteemed career path and again I completely welcome if that were the case in the future however simply decreeing that something would be very esteemed doesn't make it that way it's not fiat money so as much as I would like that I just don't believe it would work and especially I don't believe it would work right now and I think it would be subject to the same problems so can we come up with a better solution I think yes but the way to go there is to align what we want as a community with the incentives of people and not go against it because as soon as you go against it too much people will find a way around it so the first thing I want to suggest is we abolish conference publishing this weird notion that you submit your paper to this conference and then all at the same time a random process is happening and three random people give their opinion while reading your paper for a couple of minutes and then you get an accept after which your paper is there never to be revised or a reject which simply means you try again seems to be preposterous I'm sorry so people wonder yeah but how do we know when a paper is accepted who cares about acceptance who cares why can't we just switch to citations citations is a pretty good measure of how much people care about a paper and yes big names will get more citations but they do so now and they do so more effectively than ever why can't we just put our papers on archive and then run some kind of page rank algorithm over the citations such that self citations aren't worth as much I mean search engines figured out how to deliver you the most relevant search result to a query 20 years ago why can't we simply apply the same techniques to research determining this work is quite relevant this work is not that quite relevant I get it citations take time and you won't immediately know after publishing but I think that's a step we can take especially since conference publishing is also lagging like half a year behind publishing on archive during which pretty much nothing happens and then people say oh but what about peer review peer review peer review does not work peer review is a joke in machine learning okay no one cares about the reviews reviewers are a nuisance you have to get past them all the people still pretend to care that it means something that reviewers agree or disagree with you it doesn't in fact I want to get to a system where peer review starts at the moment where you publish a paper on something like archive and then never finishes for the lifetime of that paper as new knowledge comes in from the field the paper can be continuously re-examined and if the paper turns out to be really important more and more scrutiny can be applied to it seems like a much better system than simply throwing the same amount of pretty random reviewers at every paper and then giving it the stamp or not so here's what I suggest we keep something like archive but amended with a commenting function and the commenting can be pretty feature-rich so you could incorporate plots and references to other things this goes very much towards a kind of a collaboratively edited wiki but where people still put their names on things so let's say I publish a paper someone else could publish a comment which would be not less in quality than a paper it can be it can be a two-line comment it can be a full rewrite of the paper it can be an amendment so I could have published a paper and someone else could say look I've done your code on a different data set and here are the results people could then cite my paper or they could cite comments and the citations will determine the relevance the comments would also be right there on archive so every time someone goes to look at that paper they'll see the comments along with it so if the paper has a big mistake they'll basically see the comment that says hey this paper has a mistake and I can prove it right here and then they can maybe see a response to that saying no you're wrong and people can make up their own minds we could build in some kind of voting system like a stack overflow system for ranking comments but instead of making this stamp of approval thing a one-time event by a random set of people let everyone make up their own mind and let people discuss and you can even have a anonymous comments on these sites because the comments will be evaluated on what they are writing and not who it is by now of course if it does turn out that commenting will become cool after a while you can also comment non anonymously and maybe get little medals like you get on stack overflow I don't see that happening but if it does the better now as a side suggestions can we please stop publishing stuff in PDFs it's so like why do we still do this this many pages this margin and so on I get it some people still print out their papers but websites are so much nicer to look at and can be made to print adequately let's start publishing research as HTML not as PDFs so remember when I said the authors have a big incentive to not have comments on their paper this pretty much goes against that right so it is entirely conceivable that the authors will just start self hosting big company like Google could simply not publish to archive anymore they could simply publish to their own website and remove themselves from the ability for other people to comment now this can be solved technologically pretty easy by creating something like a browser plugin that if you find a piece of research anywhere it'll simply fuzzy match the title find the appropriate comments to that research as a unified set of comments across all of the internet in contrast conferences should be conferences it should be places where people come up meet up and talk about relevant issues that are happening right now if I go to a conference now most of the talks on the papers is from research that is six months old or older why don't we have conferences that are simply consisting of invited keynotes panel discussions and things that are now called workshops where we discuss current maybe unfinished research have poster sessions for many more people there's no acceptance there's no declining if there's not enough room do a lottery or something like this but make the conferences a place where science is happening and not where we flash six months old research so why is this not happening I already said that most of the incentives are actually towards the current system as much as people complain about it now conferences are slowly losing their reputations as I said because over time people will catch on to the fact that the signal being accepted at a particular conferences is more and more noisy however the system is still upheld by most PhD students for example needing a certain amount of conference accepted submissions in order to graduate so what we really need is professors and I'm calling on every professor out there to start giving out PhDs while absolutely not caring about the number of conference accepted submissions that a student has and that seems like something that's very doable because it requires individuals professors to simply change their practices with which they let people graduate so that was it for my little rant on conferences and reviewer number two please let me know what you think in the comments I value your input very much and I hope we can get to a future where conferences are conferences and research is just done on the basis of its coolness and relevance alright I'll see you bye bye
 It's review time, review time. So NeurIPS has recently released the reviews for
 submitted papers and pretty much everyone is not happy and I think the
 reason is that even though you have the reasonable reviewers of these
 conferences there is always always reviewer number two and reviewer number
 two leaves very short review says that either there are not enough experiments
 or the theory is too weak or the assumptions aren't warranted or they
 just don't like your face and that's why they give you a weak reject. Actually
 some of them think your paper is fantastic and give you a weak reject. So
 a lot of people are angry, upset, dissatisfied with the quality of the
 reviews in machine learning conferences and today I want to go look a bit into
 how this works, why this is the way that it is and what we could potentially do
 about it. So what's happening with publishing in ML? The system seems to be
 overloaded. There's so much attention in machine learning right now that there
 hasn't been a few years ago that there's a huge influx of new people wanting to
 publish in this field. That creates a lot of submissions and not enough reviewers
 to peer review these submissions. So a lot of reviewers are recruited that
 probably shouldn't be reviewers. I hear stories of undergrads being recruited as
 reviewers, people from way outside the fields, people that don't have time. So
 too many submissions, too few inexperienced and not really expert
 reviewers creates pretty much a random process and this was also shown in a few
 years ago in the then NIPS experiment where it showed that for most papers
 being accepted is pretty much a coin flip with a weighted coin. The natural
 response as an author is going to be you're going to submit even more papers.
 If it's a coin flip you can just submit whatever and there's a chance it might
 get in. Which of course only makes the problem worse. So this entire process of
 science where you submit your manuscript and then you get the reviews and then
 you try to improve it. It's completely broken because not only do you not care
 about the reviews the next set of reviewers at the next conference are
 going to be different. So no matter what you improve right now the next set of
 people will have completely different criticism. It just doesn't work like it
 is intended to work. The review process is basically just some kind of a random
 nuisance to people that they have to get through and at the same time people who
 are reviewers have every incentive to make it as hard as possible for the
 people that are submitting. So in order to analyze this I want to look at the
 incentives of the different groups in this process and kind of show how the
 incentive structure upholds this system that benefits pretty much everyone
 participating in it but creates a worse outcome for all of us. So first of all
 let's look at paper authors. What are your incentive if you're an author of a
 paper? First of all authors they want to get as many papers as possible as fast
 as possible. Now in the current conference system the fastness isn't
 really up for debate it's as fast as it is. However authors can simply upload
 their paper to archive and be as fast as they want there. Another incentive for
 authors is to have as little comments on your paper as possible because comments
 usually mean criticism and you don't want comments and especially you don't
 want public permanent comments. The good thing for authors right now is on
 archive comments aren't possible and conference reviews even if they're made
 public no one goes to look at them everyone just goes to archive. So authors
 right now are getting a pretty good deal with respect to not getting their work
 criticized. Authors are also incentivized to give as little credit to people as
 possible and again the current system is totally in favor of that. The no
 commenting on archive basically means that you can claim whatever you want and
 if someone wants to refute you they have to make a big deal out of it and
 basically write their own paper and again people will probably not find that.
 On the other hand in the conferences reviewers are supposed to detect when
 you're not giving proper credit to other people. However most reviewers don't do
 that. Going out and really looking if everything is credited properly is one
 of the most time-consuming tasks when you review a paper and most reviewers
 simply aren't going through that trouble. The only downside for most authors is
 even though all of this is pretty much in their favor a lot of them still
 require that stamp of approval that peer review accepted at a good conference. So
 their incentive is to keep submitting to conferences as many papers as possible.
 Basically count on that random process to get them accepted and after that
 they're just fine. They have the stamp of approval there's absolutely no
 requirement to revise it. There's absolutely no requirement to have other
 people comment further on the work. So I guess the complaining here right now is
 just about the noisy process and everyone complains that their particular
 paper which is at the behest of the noisy process as everyone else's paper
 got an unfair treatment in that random process which half the papers do
 probably more. The incentives in the systems are actually even bigger for
 what I call the big names. These are the big research institutions of
 companies or big name professors anyone that has some sort of reputation. People
 argue that anonymous reviewing is actually good for small authors good for
 unknown authors because it hides their identity and the big names basically
 aren't able to play their big name credit to a paper. However there's an
 easy way to know that this isn't the case. The big names are doing just fine.
 Here's the issue if you want your name to be attached to something you're gonna
 find a way to do it. People are suggesting archive blackout periods and
 whatnot anonymous submissions to archive. You have to realize that if someone wants
 to give some information to the public they are going to. In fact right now the
 big names are finding every possible way to have their names attached to things
 and massively increase their chances of getting through the anonymous peer review
 process. You got to realize if you're well connected not only do you have an
 advertising platform but you can also pretty easily find out who your area
 chairs are, who's reviewing, in which track your paper gets and so on. So allow
 me to be a little bit skeptical about the claim that we need more anonymity in
 this process. I think we need less. Second what are the incentives of the
 conferences itself? So the conference organizers they want to have a good
 reputation which basically means they want to be like a cool nightclub. Lots of
 people want to get in but they have to reject a lot of those people in order to
 make the club exclusive and have a high reputation. So conferences have every
 reason to invite everyone to submit as much as possible but then to reject as
 much as possible to make it seem like it's super hard to get in. This only
 makes the problem worse and I think the current explosion isn't really desired by
 the conferences. As the process is super noisy they're slowly losing their
 reputation that way but still the incentives aren't to lower the amount
 of submissions and increase the overall quality because that means a higher
 percentage of submissions will have to get accepted which means that the
 conference appears to be less exclusive. And lastly let's look at the reviewers
 themselves. This is the most screwed up part in the system. I have every
 incentive to be a reviewer for one of these conferences because I can write
 that on my CV. Hey I was a reviewer at big name conference and then once I am
 accepted as a reviewer I have every incentive to do absolutely nothing. In
 fact the less time I waste with this the better because I'm not getting any
 public credit. I'm anonymous right? Anonymous peer review. I'm not getting
 any reputation out of this and in fact I can only lose from accepting papers and
 I can only lose from writing detailed reviews. If I'm short and vague and I
 reject a paper not only can I not really be criticized because I'm not saying
 much it's actually in my overwhelming interest if the paper has some sort of
 big mistake and I overlook it and I accept the paper and the other reviewers
 see that mistake this looks really really bad for me even though I'm
 anonymous in the broader context it also looks bad for the area chair
 supervising me if they don't see it it looks bad for the conference if their
 area chairs don't see it so there's a massive push to not make mistakes
 however if I reject a paper that was actually good I can just say well they
 can resubmit to the next conference. So I already have a giant prior to reject a
 paper add to that that usually the papers that I review might be my
 competition and by the conference incentive of being pretty exclusive the
 more of my competition gets accepted the less I might get accepted because not
 only are there limited amount of space not formally but informally other work
 might overlap substantially with my own and therefore make it less likely that I
 get published also other work might actually criticize my work and I don't
 like that and this is a bit cynical but I'm not saying everyone does this but
 there is an incentive for you as a reviewer especially if the work is close
 to what you're doing to reject it now implement the same or a very similar
 idea and then submit to the next conference where these other authors
 also will submit and hope for the random process to just for your paper to get
 more lucky than their paper. Black planting on archive counters this a
 little bit but I'm afraid that with proposed solutions like more archive
 blackout periods more anonymity these problems will only get worse maybe some
 people don't realize this but as a reviewer it's really easy for me to
 reject a paper I can almost always find reasons to reject a paper if it's a
 theory paper I can ask for experiments if it's an experimental paper I can ask
 for more experiments why didn't you test that data set why didn't you compare
 against this method why are your assumptions so strong they're never
 guaranteed in practice is the problem even relevant your theory is too weak
 have you looked at this other special case and if I really want to I can just
 ask many many many questions not even criticisms just many questions and I know
 the authors just have a one-page rebuttal they can never answer all my
 questions if I do that and then I can simply argue the authors failed to
 address all my questions properly so you might be asking why do some reviewers
 actually do a good job and that is I believe really a lot to do with good
 will most people are actually well intended most people actually want to do
 a good job in reviewing have the ethos of science and do take the time to do
 the reviews even though they're incentivized to do them badly even
 though they're incentivized to reject papers a lot of people still do a good
 job however reviewer number two usually doesn't and it only needs a very few
 reviewer number twos to make the field a whole lot worse now there's a question
 to be said aren't we all a bit reviewer number two have you ever written a
 review that the authors might think is completely unreasonable and while there
 is some truth to that argument I definitely know that there are
 differences in reviews in fact I've heard people brag about writing two line
 reviews where the second line is you didn't cite and compare to my own work
 and then laugh about that so goodwill won't carry us all the way if the
 incentive structure is bad and I believe most of this is because we've taken out
 the reputation game out of the review process in smaller fields of science it
 used to be that the journal editors knew the reviewers and their reputation at
 least towards the journal editor was on the line for all of future if they did a
 bad job right now everything's so big so anonymous people hardly remember the
 names of their co-reviewers no reputation is being damaged by bad
 reviews and that's how we get here so of course I'm not the first one to observe
 these problems so many people have proposed solutions and most of these
 solutions fall into the basis of what I would call a C based methods which is
 basically where someone evaluates the reviews while the reviews remain
 anonymous and that someone is usually the area chair so right now the area
 chair can already decide that a reviewer is really bad and then the reviewer will
 not be invited to review the next time around I just want to point out the
 irony of the situation conferences nowadays have so little reviewers that
 they require every author to be a reviewer but then your punishment for
 writing bad reviews is that you won't be invited to be a reviewer the next time
 around I mean can you make a better point that the system is failing of
 course the problem with all a C based methods is that you're basically moving
 a problem that has everything to do with people being unaccountable noisy not
 expertly having no time and every single incentive to do as little as possible
 you transfer that problem to even less people that have even less time that
 have even more stress that have an even broader view and topic area and are
 single people instead of three or four people so it's even more noisy if
 anything like this is implemented you'll just instead of seeing complaints about
 bad reviews in addition you will also see complaints about bad a C's that will
 certainly not make the problem any less in fact I would argue any a C based
 solution will make the problem worse other solutions are what I called
 payment based solutions like give the reviewers money to review I don't see
 how that fixes the incentive for you to reject anything you just might write it
 in a bit more eloquent style also as soon as you bring money into the game
 that automatically excludes a lot of people depending on how you do it that
 aren't as affluent which is certainly something we don't want as a community
 other people are pointing to things like open review which I agree is a better
 system however it is still anonymous so the same incentives exist and it is
 still a conference where you get a stamp of accept or reject and once it's
 accepted no one cares about the reviews anymore in fact in open review you can
 write as much text as you want so the a C's are even more overloaded with lots
 of text to make their decisions so something I want to highlight is a
 thread by Thomas G Dietrich on Twitter where he basically suggests some sort
 of a wiki and sort of a collaborative research wiki where you'd have a set of
 senior authors that basically maintain that wiki that do a first check of
 papers and kind of match them against the wiki of what's already known I
 won't go through that here I will link it and I definitely advise you to read
 it because it's very interesting proposal it's a sort of utopian dream I
 would actually welcome if we all work together on increasing the knowledge of
 mankind in a wiki style way however I think lots of people want their names
 attached to things and even if you do what Thomas suggests and basically have
 people write papers and then the editors integrate that into the wiki it is not
 clear how that system where the editors clearly need to be senior and
 experienced could deal any better with the explosion of research that we're
 dealing right now they would be as overloaded as the current system plus
 who's gonna be an editor Thomas says becoming an editor would be a very
 esteemed career path and again I completely welcome if that were the case
 in the future however simply decreeing that something would be very esteemed
 doesn't make it that way it's not fiat money so as much as I would like that I
 just don't believe it would work and especially I don't believe it would work
 right now and I think it would be subject to the same problems so can we
 come up with a better solution I think yes but the way to go there is to align
 what we want as a community with the incentives of people and not go against
 it because as soon as you go against it too much people will find a way around
 it so the first thing I want to suggest is we abolish conference publishing this
 weird notion that you submit your paper to this conference and then all at the
 same time a random process is happening and three random people give their
 opinion while reading your paper for a couple of minutes and then you get an
 accept after which your paper is there never to be revised or a reject which
 simply means you try again seems to be preposterous I'm sorry so people wonder
 yeah but how do we know when a paper is accepted who cares about acceptance who
 cares why can't we just switch to citations citations is a pretty good
 measure of how much people care about a paper and yes big names will get more
 citations but they do so now and they do so more effectively than ever why can't
 we just put our papers on archive and then run some kind of page rank
 algorithm over the citations such that self citations aren't worth as much I
 mean search engines figured out how to deliver you the most relevant search
 result to a query 20 years ago why can't we simply apply the same
 techniques to research determining this work is quite relevant this work is not
 that quite relevant I get it citations take time and you won't immediately know
 after publishing but I think that's a step we can take especially since
 conference publishing is also lagging like half a year behind publishing on
 archive during which pretty much nothing happens and then people say oh but what
 about peer review peer review peer review does not work peer review is a joke in
 machine learning okay no one cares about the reviews reviewers are a nuisance you
 have to get past them all the people still pretend to care that it means
 something that reviewers agree or disagree with you it doesn't in fact I
 want to get to a system where peer review starts at the moment where you
 publish a paper on something like archive and then never finishes for the
 lifetime of that paper as new knowledge comes in from the field the paper can be
 continuously re-examined and if the paper turns out to be really important
 more and more scrutiny can be applied to it seems like a much better system than
 simply throwing the same amount of pretty random reviewers at every paper
 and then giving it the stamp or not so here's what I suggest we keep something
 like archive but amended with a commenting function and the commenting
 can be pretty feature-rich so you could incorporate plots and references to
 other things this goes very much towards a kind of a collaboratively edited wiki
 but where people still put their names on things so let's say I publish a paper
 someone else could publish a comment which would be not less in quality than
 a paper it can be it can be a two-line comment it can be a full rewrite of the
 paper it can be an amendment so I could have published a paper and someone else
 could say look I've done your code on a different data set and here are the
 results people could then cite my paper or they could cite comments and the
 citations will determine the relevance the comments would also be right there
 on archive so every time someone goes to look at that paper they'll see the
 comments along with it so if the paper has a big mistake they'll basically see
 the comment that says hey this paper has a mistake and I can prove it right here
 and then they can maybe see a response to that saying no you're wrong and
 people can make up their own minds we could build in some kind of voting
 system like a stack overflow system for ranking comments but instead of making
 this stamp of approval thing a one-time event by a random set of people let
 everyone make up their own mind and let people discuss and you can even have a
 anonymous comments on these sites because the comments will be evaluated on
 what they are writing and not who it is by now of course if it does turn out
 that commenting will become cool after a while you can also comment non
 anonymously and maybe get little medals like you get on stack overflow I don't
 see that happening but if it does the better now as a side suggestions can we
 please stop publishing stuff in PDFs it's so like why do we still do this
 this many pages this margin and so on I get it some people still print out their
 papers but websites are so much nicer to look at and can be made to print
 adequately let's start publishing research as HTML not as PDFs so
 remember when I said the authors have a big incentive to not have comments on
 their paper this pretty much goes against that right so it is entirely
 conceivable that the authors will just start self hosting big company like
 Google could simply not publish to archive anymore they could simply
 publish to their own website and remove themselves from the ability for other
 people to comment now this can be solved technologically pretty easy by creating
 something like a browser plugin that if you find a piece of research anywhere
 it'll simply fuzzy match the title find the appropriate comments to that
 research as a unified set of comments across all of the internet in contrast
 conferences should be conferences it should be places where people come up
 meet up and talk about relevant issues that are happening right now if I go to
 a conference now most of the talks on the papers is from research that is six
 months old or older why don't we have conferences that are simply consisting
 of invited keynotes panel discussions and things that are now called workshops
 where we discuss current maybe unfinished research have poster sessions
 for many more people there's no acceptance there's no declining if
 there's not enough room do a lottery or something like this but make the
 conferences a place where science is happening and not where we flash six
 months old research so why is this not happening I already said that most of
 the incentives are actually towards the current system as much as people
 complain about it now conferences are slowly losing their reputations as I
 said because over time people will catch on to the fact that the signal being
 accepted at a particular conferences is more and more noisy however the system
 is still upheld by most PhD students for example needing a certain amount of
 conference accepted submissions in order to graduate so what we really need is
 professors and I'm calling on every professor out there to start giving out
 PhDs while absolutely not caring about the number of conference accepted
 submissions that a student has and that seems like something that's very doable
 because it requires individuals professors to simply change their
 practices with which they let people graduate so that was it for my little
 rant on conferences and reviewer number two please let me know what you think in
 the comments I value your input very much and I hope we can get to a future
 where conferences are conferences and research is just done on the basis of
 its coolness and relevance alright I'll see you bye bye