State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication
deep learning machine learning nlp natural language processing arxiv attention peer review automate distributed scalable neurips score objective
Peer Review is outdated and ineffective. SOAR is a new and revolutionary way to distribute scientific reviewing and scale to the new age of faster, better and more significant research.

https://arxiv.org/abs/2003.14415

Abstract:
Peer review forms the backbone of modern scientific manuscript evaluation. But after two hundred and eighty-nine years of egalitarian service to the scientific community, does this protocol remain fit for purpose in 2020? In this work, we answer this question in the negative (strong reject, high confidence) and propose instead State-Of-the-Art Review (SOAR), a neoteric reviewing pipeline that serves as a 'plug-and-play' replacement for peer review. At the heart of our approach is an interpretation of the review process as a multi-objective, massively distributed and extremely-high-latency optimisation, which we scalarise and solve efficiently for PAC and CMT-optimal solutions. We make the following contributions: (1) We propose a highly scalable, fully automatic methodology for review, drawing inspiration from best-practices from premier computer vision and machine learning conferences; (2) We explore several instantiations of our approach and demonstrate that SOAR can be used to both review prints and pre-review pre-prints; (3) We wander listlessly in vain search of catharsis from our latest rounds of savage CVPR rejections.

Authors: Samuel Albanie, Jaime Thewmore, Robert McCraith, Joao F. Henriques

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi everyone. Today we're looking at state-of-the-art reviewing a radical proposal to improve scientific publication. This has been on my mind for a while. The review process for modern science, especially machine learning, is just broken. I've spoken numerous times about the fact that we need to replace it with a better system. Samuel Albany at Al have actually come up with such a system and we're going to explore it today. I am a big fan of this work and I'm 100% on board with this. They basically say peer review forms the backbone of modern scientific manuscript evaluation. If you don't know what peer review is in machine learning right now, if you have some genius idea, so here is your idea, that's a light bulb by the way, you write it up into an eight-page PDF. Yes, it must be a PDF and yes, it must be eight pages. You submit it to a conference, which is some kind of... So you submit it to be accepted in a conference proceeding. And if conference organizers, of course, they're just a bunch of people, they can't review these 1,000 million applications that come by themselves. So what they do is they recruit experts, which are called peers. So peers are other people. These are called peers and they have usually written up their own papers and they can critique each other's paper and they decide what gets accepted and what doesn't. Now, I've spoken numerous times of how this is super noisy right now. They're way, way, they're not enough peers, they're not experienced enough. So whether or not your particular idea gets accepted is extremely dependent on probability, on a coin flip usually. And it's just overloaded and just makes no sense. And they ask the same question, is this fit for purpose in 2020? And we need to replace it and I support. So they, you can already see they kind of want to automate this away with their state of the art review score and the score will be an out of 10 score that can be integrated into something like archive. And, you know, display right away. So they have some requirements to this new system. What should it be done? It should have the ability to scale. Very important. Our current review system doesn't have this, right? Our current review system relies on other humans reviewing your paper. And that means that the reviewers need to scale with the amount of papers, which just isn't the case currently. So a new review system must have the ability to scale. Right. And then, you know, automating the reviews away or scaling it up in a distributed fashion does this. Speed. Yes, because right now, if I submit my manuscript for review, it takes them months to review it. And our science progress is faster than that. So a speedy, more speedy version of peer review is definitely required. And then consistency. And this is the most shocking part, right? There is a the grand 2014 NURIPS experiment, which concluded that 57 percent of papers accepted by one committee were rejected by another committee and vice versa. Reviewing the exact same paper, different committees came to completely different conclusions to an astounding degree. So basically, you're flipping a coin of whether or not your paper gets accepted or not. And I think this is just not acceptable. And so they propose these three things, speed, scale, consistency, and their new method certainly has this. Now, let's jump down here where they introduce this state of the art reviewing SOAR. So they say, OK, the quality of a scientific work can be judged along three axes, efficacy, significance and novelty. So there are these three pillars, right? Efficacy, which means is is kind of how how effective is your work in achieving the goal in machine learning? That's usually to train some good classifier or something like this. Then the other one, sorry, is significance, right? Significance is how relevant is what you've done to the to the field. Right. And the third one is novelty. So, you know, in your scientific work should be an original contribution to the knowledge of mankind and therefore it should be novel. Right. So the more of these three things you have, of course, the higher your score should be. And here in the middle is where the highest scores should be. So imagine this is kind of a landscape. And so you want to grade papers along these three axes. But they have a pretty good method of of of assessing these in an automated fashion. So, first of all, assessing efficacy, efficacy, they say, is best assessed by determining if the proposed method achieves a new state of the art. I think that's not really I don't think you can really doubt this. I mean, this this is this is kind of the gold standard of of whether your paper is effective, is whether or not it achieves state of the art. I mean, I it might be a bit of a controversial opinion, but if a paper doesn't achieve a state of the art, it's you know, why? Why do you even care? Like no one cares. So from they say from an implementation perspective, they can they can use they can kind of abuse a fact of the current research environment is that you don't actually have to review this yourself. But the authors themselves can be relied upon to state this repeatedly in the text. Right. And this this is important. So the authors will state that they have state of the art many, many times in the text if they have actually achieved it. If they haven't achieved it or not so sure about it, they probably won't repeat it as many times. But this is can be can kind of abuse now to distribute it. Basically, you don't imagine now these these all of these reviewers. They don't they don't have to do this work anymore. They can just distribute to all the authors of their own papers, right? Because the authors in the text by the way, the text is structures is kind of an NLP approach to reviewing kind of NLP mixed with game theory. Right. So the other authors themselves if they have state of the art, you have to do some stemming and stuff, but they will put that into the text a lot. So it's a bit controversial, but the the authors here propose to simply count the number of word occurrences of state of the art case and sensitive very important in the text, right? It stands to reason that a higher state of the art count is preferable. Of course. All right. So the second thing so this might be a bit controversial. The second thing significance and they now make the claim significance is measured by efficacy. So they simply the efficacy term. So if your paper is effective at achieving its goal, you can also say it's significant for the community because again, significance should like if you have state of the art, then your paper is significant. If you do not have state of the art, then your paper is obviously not significant because why should it matter if you don't have state of the art in a given task? It's useless. All right. So we weigh it twice. That's pretty good. And then novelty now here they take much of the same approach. They say the authors probably state this. So how much they use the word novel in their manuscript will dictate. So here they say, okay, they novel. Wow, this is failing me. Hello. How much they use the word novel in the text will probably be an indication. I don't think so though. They do do the smart thing of they include the works. They include the related work section from this. Sorry, they exclude the related work section. They say we make the key observation that individuals best play to make the judgment are the authors themselves since they have likely read at least one of the works cited in the bibliography. I don't agree here. I think a better method would be to simply count the number of references and the lower the amount of references to related work, the higher the novelty. Because if you think, if these are current papers and your paper is here, you'll have a lot of related work. So it's not as novel. If you're way out here, you'll have maybe one or two related works. So it's way more novel if you have less references. So this would be my criticism of this. So this novelty thing here, I think this term should be replaced by a graph centrality measure or simply a count of how many references you have would be enough. All right, so they define their score. Their score, as we saw, is the SOTA term weighted twice. A geometric mean between that and the novelty term, which I've criticized. They add the suffix out of 10 because out of 10 score is pretty interpretable. So they divide by 10 here. So yeah, they say that here. We attach a suffix out of 10 because that's easy to interpret. And as you saw in the kind of archive implementation right here, sorry, this will be then easy to integrate right here. So they even give code, right? They give code in the paper themselves of how to implement this. It's pretty easy. And I think, yeah, even though it's quite a short paper, it's thorough and it's a good new method. And I think this could revolutionize publishing. And they even, so as a bit of a bonus, they give the official pronunciation of state of the art reviewing, which is something like state of the art reviewing pretty smooth. And yeah, with that, I hope you enjoyed this. And if the authors could just be a little more subtle next time, that would be great. And I guess you'd have to go. Yeah, nothing more. Bye.
 Hi everyone. Today we're looking at state-of-the-art reviewing a radical proposal to improve scientific publication.
 This has been on my mind for a while. The review process for modern science, especially machine learning, is just broken.
 I've spoken numerous times about the fact that we need to replace it with a better system.
 Samuel Albany at Al have actually come up with such a system and we're going to explore it today.
 I am a big fan of this work and I'm 100% on board with this.
 They basically say peer review forms the backbone of modern scientific manuscript evaluation.
 If you don't know what peer review is in machine learning right now, if you have some genius idea,
 so here is your idea, that's a light bulb by the way, you write it up into an eight-page PDF.
 Yes, it must be a PDF and yes, it must be eight pages. You submit it to a conference, which is some kind of...
 So you submit it to be accepted in a conference proceeding.
 And if conference organizers, of course, they're just a bunch of people, they can't review these 1,000 million applications that come by themselves.
 So what they do is they recruit experts, which are called peers. So peers are other people.
 These are called peers and they have usually written up their own papers and they can critique each other's paper
 and they decide what gets accepted and what doesn't. Now, I've spoken numerous times of how this is super noisy right now.
 They're way, way, they're not enough peers, they're not experienced enough.
 So whether or not your particular idea gets accepted is extremely dependent on probability, on a coin flip usually.
 And it's just overloaded and just makes no sense. And they ask the same question, is this fit for purpose in 2020?
 And we need to replace it and I support. So they, you can already see they kind of want to automate this away
 with their state of the art review score and the score will be an out of 10 score that can be integrated into something like archive.
 And, you know, display right away. So they have some requirements to this new system.
 What should it be done? It should have the ability to scale. Very important. Our current review system doesn't have this, right?
 Our current review system relies on other humans reviewing your paper.
 And that means that the reviewers need to scale with the amount of papers, which just isn't the case currently.
 So a new review system must have the ability to scale. Right.
 And then, you know, automating the reviews away or scaling it up in a distributed fashion does this.
 Speed. Yes, because right now, if I submit my manuscript for review, it takes them months to review it.
 And our science progress is faster than that. So a speedy, more speedy version of peer review is definitely required.
 And then consistency. And this is the most shocking part, right?
 There is a the grand 2014 NURIPS experiment, which concluded that
 57 percent of papers accepted by one committee were rejected by another committee and vice versa. Reviewing the exact same paper,
 different committees came to completely different conclusions to an astounding degree.
 So basically, you're flipping a coin of whether or not your paper gets accepted or not.
 And I think this is just not acceptable.
 And so they propose these three things, speed, scale, consistency, and their new method certainly has this.
 Now, let's jump down here where they introduce this state of the art reviewing SOAR.
 So they say, OK, the quality of a scientific work can be judged along three axes, efficacy, significance and novelty.
 So there are these three pillars, right?
 Efficacy, which means is is kind of how how effective is your work in achieving the goal in machine learning?
 That's usually to train some good classifier or something like this.
 Then the other one, sorry, is significance, right?
 Significance is how relevant is what you've done to the to the field.
 Right. And the third one is novelty.
 So, you know, in your scientific work should be an original contribution to the knowledge of mankind and therefore it should be novel.
 Right. So the more of these three things you have, of course,
 the higher your score should be. And here in the middle is where the highest scores should be.
 So imagine this is kind of a landscape. And so you want to grade papers along these three axes.
 But they have a pretty good method of of of assessing these in an automated fashion.
 So, first of all, assessing efficacy, efficacy, they say,
 is best assessed by determining if the proposed method achieves a new state of the art.
 I think that's not really I don't think you can really doubt this.
 I mean, this this is this is kind of the gold standard of of whether your paper is effective,
 is whether or not it achieves state of the art.
 I mean, I it might be a bit of a controversial opinion,
 but if a paper doesn't achieve a state of the art, it's you know, why?
 Why do you even care? Like no one cares.
 So from they say from an implementation perspective,
 they can they can use they can kind of abuse a fact of the current research environment is that you don't actually have to review this yourself.
 But the authors themselves can be relied upon to state this repeatedly in the text.
 Right. And this this is important.
 So the authors will state that they have state of the art many,
 many times in the text if they have actually achieved it.
 If they haven't achieved it or not so sure about it,
 they probably won't repeat it as many times.
 But this is can be can kind of abuse now to distribute it.
 Basically, you don't imagine now these these all of these reviewers.
 They don't they don't have to do this work anymore.
 They can just distribute to all the authors of their own papers,
 right? Because the authors in the text by the way,
 the text is structures is kind of an NLP approach to reviewing kind of NLP mixed with game theory.
 Right. So the other authors themselves if they have state of the art,
 you have to do some stemming and stuff, but they will put that into the text a lot.
 So it's a bit controversial,
 but the the authors here propose to simply count the number of word occurrences of state of the art case
 and sensitive very important in the text, right?
 It stands to reason that a higher state of the art count is preferable.
 Of course.
 All right.
 So the second thing so this might be a bit controversial.
 The second thing significance and they now make the claim significance is measured by efficacy.
 So they simply the efficacy term.
 So if your paper is effective at achieving its goal,
 you can also say it's significant for the community because again,
 significance should like if you have state of the art,
 then your paper is significant.
 If you do not have state of the art, then your paper is obviously not significant
 because why should it matter if you don't have state of the art in a given task?
 It's useless.
 All right.
 So we weigh it twice.
 That's pretty good.
 And then novelty now here they take much of the same approach.
 They say the authors probably state this.
 So how much they use the word novel in their manuscript will dictate.
 So here they say, okay, they novel.
 Wow, this is failing me.
 Hello.
 How much they use the word novel in the text will probably be an indication.
 I don't think so though.
 They do do the smart thing of they include the works.
 They include the related work section from this.
 Sorry, they exclude the related work section.
 They say we make the key observation that individuals best play to make the judgment
 are the authors themselves since they have likely read at least one of the works
 cited in the bibliography.
 I don't agree here.
 I think a better method would be to simply count the number of references
 and the lower the amount of references to related work, the higher the novelty.
 Because if you think, if these are current papers and your paper is here,
 you'll have a lot of related work.
 So it's not as novel.
 If you're way out here, you'll have maybe one or two related works.
 So it's way more novel if you have less references.
 So this would be my criticism of this.
 So this novelty thing here, I think this term should be replaced by a graph
 centrality measure or simply a count of how many references you have would be enough.
 All right, so they define their score.
 Their score, as we saw, is the SOTA term weighted twice.
 A geometric mean between that and the novelty term, which I've criticized.
 They add the suffix out of 10 because out of 10 score is pretty interpretable.
 So they divide by 10 here.
 So yeah, they say that here.
 We attach a suffix out of 10 because that's easy to interpret.
 And as you saw in the kind of archive implementation right here,
 sorry, this will be then easy to integrate right here.
 So they even give code, right?
 They give code in the paper themselves of how to implement this.
 It's pretty easy.
 And I think, yeah, even though it's quite a short paper,
 it's thorough and it's a good new method.
 And I think this could revolutionize publishing.
 And they even, so as a bit of a bonus,
 they give the official pronunciation of state of the art reviewing,
 which is something like state of the art reviewing pretty smooth.
 And yeah, with that, I hope you enjoyed this.
 And if the authors could just be a little more subtle next time,
 that would be great.
 And I guess you'd have to go.
 Yeah, nothing more.
 Bye.