Research Conference ICML drops their acceptance rate | Area Chairs instructed to be more picky
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper icml peer review machine learning conference icml conference icml submission icml paper accepted how to write machine learning papers how to publish a paper how to publish in machine learning how to do a phd in machine learning deep learning conference machine learning research conference icml acceptance rate icml submissions icml area chairs machine learning news
#icml #machinelearning #conference

In a controversial move, ICML Area Chairs were instructed to raise the bar on acceptance to drop the acceptance rate by 10% from the previous trajectory. This raises a lot of questions about the pains of an academic peer review system under the load of an exponentially increasing field of study. Who draws the short stick? Usually not the big corporations.

References:
https://www.reddit.com/r/MachineLearning/comments/n243qw/d_icml_conference_we_plan_to_reduce_the_number_of/
https://twitter.com/tomgoldsteincs/status/1388156022112624644
https://twitter.com/ryan_p_adams/status/1388164670410866692
https://github.com/lixin4ever/Conference-Acceptance-Rate

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Good morning, I hope you had a good night's sleep. It's just another day where the review system in machine learning is completely and utterly broken this time courtesy of the ICML chairs, apparently notifying the senior area chairs to reduce the number of accepted submissions by about 10%. According to current meta review statistics, we need to raise the acceptance bar. Also saying we plan to reduce the number of accepted papers, please work with your senior area chair to raise the bar area chairs and senior area chairs do not have to accept a paper only because there is nothing wrong with it. So the ICML conference is trying to raise the bar on scientific publication in their venue by just accepting a little bit less papers than they would do according to current trajectory of the review process. ICML currently is in the post review post rebuttal process where the actual acceptance decisions are made. Now, why is this important? This is important because there are only about three or four large conferences in machine learning each year depending on your subfield bit more or even a bit less. For many places, if you want to get a PhD, if you want to get tenure, if you want to achieve anything in academia, you need to publish papers at those venues. And given that the field is exploding currently getting a paper there is quite difficult acceptance rates have been dropping steadily in the past few years, though you can see the number of accepted papers has actually risen. This is a consequence of the exponential growth of the machine learning field. Now there's a growing concern that the review process isn't really good. And what gets published and what doesn't get published is just kind of a wash in the noisy process, which is true. I've made quite a number of videos about the really flawed review process in machine learning. Essentially, here is what we know, if your paper is really good, then it's going to get accepted very probably, you might get unlucky, but with high probability, it's going to get there. If your paper is really bad, also with a high probability, it's going to get rejected. However, for most papers, which aren't extremely good, which aren't extremely bad, there's just this middle area, most papers fall into this middle area. And it's really a roll of a dice, you get some reviewers, they might know what they're talking about, they might not know what they're talking about, they have their favorite data set, you didn't evaluate on it, they reject or they weak accept because they just don't want to deal with your rebuttal. It's an all around fun process, but it can ruin your life. And for a conference such as ICML, it is important that it keeps up its reputation for only publishing the best papers and really good scientific results. So by reducing the acceptance rate, what they'll do is they'll put more focus on the really good papers that stand out, which can be interpreted as a good thing, because ultimately, the really good papers will still stay while some of the borderline papers will drop out. That gives you a stronger signal that whatever comes from this conference is a valuable scientific publication. On the other hand, you can say given how noisy that review process is, you simply compress a little bit the amount of people that draw a lucky lottery ticket. And given that the field is growing, and there is huge pressure on people to publish, and also the fact that large corporations throw extreme amounts of money of getting papers published at these conferences, weeding out the academics that don't have as much resources, it is a bit of a controversial decision. Essentially, reviewers and area chairs are even more incentivized to just find anything wrong with a paper and rejected because of it. And the downside of that is that if you don't have as much resources to train on every data set, you're probably going to be out much more likely. And also if you have some really cool idea that just doesn't work yet quite well doesn't beat state of the art yet, but is quite interesting, also very probably you're not going to get there. So while the optimist might see a stronger signal for an acceptance rating at that conference, and just higher quality output, and the pessimist might see the noisy process and say, well, what is it all worth? It doesn't mean anything to get accepted anyway. And now it's just less papers that do. And also large companies are going to dominate the field. And also academics are going to draw the short stick. The optimist and the pessimist are no match for the PhD student. See, what they seem to be doing right here is specify the acceptance their target in percent, which means number of accepted papers divided by number of submitted papers. I hope you see where this is going. The target acceptance rate in the eyes of the conference means that the numerator should be smaller. However, you can reach that same acceptance rate by just making the denominator larger. Now hypothetically, if just everyone would submit more papers, we could drop the acceptance rate, but also raise the chances that our actual papers are going to get in. Now, in this hypothetical scenario, I would not be advocating for submitting fake papers or just empty PDFs. But you might have some papers in the drawer, like this beauty right here that I wrote back in I don't know when, where I designed a method to defend against black box model theft attacks, which I thought was pretty smart. But honestly, it needs a lot of work to actually make it work. And I just did not bother. It's an archive right now. But even though I am not happy with it as it is, it is certainly better than a lot of stuff that I've seen submitted to ICML that I've read as a reviewer, and even some stuff that actually got accepted at the end. So compared to that, I don't see a reason why this should not be worthy. So you, my friend, are going to ICML next year. How about that? Of course, all just a hypothetical. I'm not advocating for you to mess with a system that's clearly broken and needs to be renewed. And we should reinvent the whole thing. However, it's fun to think about. If you have some thoughts on hypothetical scenarios or stories about how your papers got rejected that we all love to tell, tell me in the comments and see you next time.
 Good morning, I hope you had a good night's sleep. It's just another day where the review system
 in machine learning is completely and utterly broken this time courtesy of the ICML chairs,
 apparently notifying the senior area chairs to reduce the number of accepted submissions
 by about 10%. According to current meta review statistics, we need to raise the acceptance bar.
 Also saying we plan to reduce the number of accepted papers, please work with your senior
 area chair to raise the bar area chairs and senior area chairs do not have to accept a paper only
 because there is nothing wrong with it. So the ICML conference is trying to raise the bar on
 scientific publication in their venue by just accepting a little bit less papers than they
 would do according to current trajectory of the review process. ICML currently is in the post
 review post rebuttal process where the actual acceptance decisions are made. Now, why is this
 important? This is important because there are only about three or four large conferences in
 machine learning each year depending on your subfield bit more or even a bit less. For many
 places, if you want to get a PhD, if you want to get tenure, if you want to achieve anything in
 academia, you need to publish papers at those venues. And given that the field is exploding
 currently getting a paper there is quite difficult acceptance rates have been dropping steadily in
 the past few years, though you can see the number of accepted papers has actually risen. This is a
 consequence of the exponential growth of the machine learning field. Now there's a growing concern
 that the review process isn't really good. And what gets published and what doesn't get published is
 just kind of a wash in the noisy process, which is true. I've made quite a number of videos about
 the really flawed review process in machine learning. Essentially, here is what we know,
 if your paper is really good, then it's going to get accepted very probably, you might get unlucky,
 but with high probability, it's going to get there. If your paper is really bad, also with
 a high probability, it's going to get rejected. However, for most papers, which aren't extremely
 good, which aren't extremely bad, there's just this middle area, most papers fall into this middle
 area. And it's really a roll of a dice, you get some reviewers, they might know what they're
 talking about, they might not know what they're talking about, they have their favorite data set,
 you didn't evaluate on it, they reject or they weak accept because they just don't want to deal
 with your rebuttal. It's an all around fun process, but it can ruin your life. And for a conference
 such as ICML, it is important that it keeps up its reputation for only publishing the best papers
 and really good scientific results. So by reducing the acceptance rate, what they'll do is they'll
 put more focus on the really good papers that stand out, which can be interpreted as a good thing,
 because ultimately, the really good papers will still stay while some of the borderline papers
 will drop out. That gives you a stronger signal that whatever comes from this conference is a
 valuable scientific publication. On the other hand, you can say given how noisy that review
 process is, you simply compress a little bit the amount of people that draw a lucky lottery ticket.
 And given that the field is growing, and there is huge pressure on people to publish, and also the
 fact that large corporations throw extreme amounts of money of getting papers published at these
 conferences, weeding out the academics that don't have as much resources, it is a bit of a
 controversial decision. Essentially, reviewers and area chairs are even more incentivized to just
 find anything wrong with a paper and rejected because of it. And the downside of that is that
 if you don't have as much resources to train on every data set, you're probably going to be
 out much more likely. And also if you have some really cool idea that just doesn't work yet quite
 well doesn't beat state of the art yet, but is quite interesting, also very probably you're not
 going to get there. So while the optimist might see a stronger signal for an acceptance rating at
 that conference, and just higher quality output, and the pessimist might see the noisy process and
 say, well, what is it all worth? It doesn't mean anything to get accepted anyway. And now it's just
 less papers that do. And also large companies are going to dominate the field. And also academics
 are going to draw the short stick. The optimist and the pessimist are no match for the PhD student.
 See, what they seem to be doing right here is specify the acceptance their target in percent,
 which means number of accepted papers divided by number of submitted papers.
 I hope you see where this is going. The target acceptance rate in the eyes of the conference
 means that the numerator should be smaller. However, you can reach that same acceptance rate
 by just making the denominator larger. Now hypothetically, if just everyone would submit
 more papers, we could drop the acceptance rate, but also raise the chances that our actual papers
 are going to get in. Now, in this hypothetical scenario, I would not be advocating for submitting
 fake papers or just empty PDFs. But you might have some papers in the drawer, like this beauty
 right here that I wrote back in I don't know when, where I designed a method to defend against
 black box model theft attacks, which I thought was pretty smart. But honestly, it needs a lot of work
 to actually make it work. And I just did not bother. It's an archive right now. But even
 though I am not happy with it as it is, it is certainly better than a lot of stuff that I've
 seen submitted to ICML that I've read as a reviewer, and even some stuff that actually got accepted at
 the end. So compared to that, I don't see a reason why this should not be worthy. So you, my friend,
 are going to ICML next year. How about that? Of course, all just a hypothetical. I'm not advocating
 for you to mess with a system that's clearly broken and needs to be renewed. And we should
 reinvent the whole thing. However, it's fun to think about. If you have some thoughts on hypothetical
 scenarios or stories about how your papers got rejected that we all love to tell, tell me in
 the comments and see you next time.