Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper neurips nips nips experiment peer reviw conference review reviewer machine learning reviewer ml conference review subjectivity in peer review reviewer opinions science review science peer review peer review fail
#neurips #peerreview #nips

The peer-review system at Machine Learning conferences has come under much criticism over the last years. One major driver was the infamous 2014 NeurIPS experiment, where a subset of papers were given to two different sets of reviewers. This experiment showed that only about half of all accepted papers were consistently accepted by both committees and demonstrated significant influence of subjectivity. This paper revisits the data from the 2014 experiment and traces the fate of accepted and rejected papers during the 7 years since, and analyzes how well reviewers can assess future impact, among other things.

OUTLINE:
0:00 - Intro & Overview
1:20 - Recap: The 2014 NeurIPS Experiment
5:40 - How much of reviewing is subjective?
11:00 - Validation via simulation
15:45 - Can reviewers predict future impact?
23:10 - Discussion & Comments

Paper: https://arxiv.org/abs/2109.09774
Code: https://github.com/lawrennd/neurips2014/

Abstract:
In this paper we revisit the 2014 NeurIPS experiment that examined inconsistency in conference peer review. We determine that 50% of the variation in reviewer quality scores was subjective in origin. Further, with seven years passing since the experiment we find that for accepted papers, there is no correlation between quality scores and impact of the paper as measured as a function of citation count. We trace the fate of rejected papers, recovering where these papers were eventually published. For these papers we find a correlation between quality scores and impact. We conclude that the reviewing process for the 2014 conference was good for identifying poor papers, but poor for identifying good papers. We give some suggestions for improving the reviewing process but also warn against removing the subjective element. Finally, we suggest that the real conclusion of the experiment is that the community should place less onus on the notion of top-tier conference publications when assessing the quality of individual researchers. For NeurIPS 2021, the PCs are repeating the experiment, as well as conducting new ones.

Authors: Corinna Cortes, Neil D. Lawrence

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at inconsistency in conference peer review, revisiting the 2014 NeurIPS experiment by Corina Cortes and Neil D. Lawrence, which were actually the chairs of the 2014 NeurIPS conference. So they are going to have access to some data that the rest of us sadly don't have access to. So it allows them to make pretty cool research on how conference reviewing works and whether or not it actually can determine the quality of a paper or how much of it is just random subjective reviewer decisions. Now this paper particularly here takes up the papers that were subject to the 2014 NeurIPS experiment and tracks them over time. So it looks at the papers that were submitted, how did they perform in the subsequent years, meaning how many citations that they accumulate, both for the accepted and for the rejected papers. And they find some pretty interesting results right here. So we'll dive into this. The paper is not too long and the conclusions are fairly straightforward. I still think it's really cool that people actually follow up on this work. So for those of you who don't know, the 2014 NeurIPS experiment, that is the wrong color, the 2014 NeurIPS experiment was an experiment in assessing how much of review of conference review is random essentially. So what you did was, and I think they have a little section about this here. So they selected about 10% of the submissions. These were 170 papers and these would undergo review by two separate committees. So whereas usually you have a paper that goes into a review, let's call that a committee, which is a bunch of reviewers and an area chair and they make the decisions of whether to accept or to reject. And yeah, at the end you have a decision. So in this experiment, you would take a paper, you would actually give it to two different committees, committee one and committee two. Committee one would only be selected from kind of one half of the reviewer pool and committee two would only be selected from the other half. These were random assignments to the two pools and also the papers who participated were randomly selected. So each of these committees would reach their own decision, accept or reject. And of course, the interesting part is how many of those agree or how many of those disagree with each other. And by the way, the paper would be accepted finally, if the max, so if either of the committees would accept the paper. And if I recall correctly, this year's NeurIPS conference actually repeats that experiment from 2014. So we're going to have another data point in hopefully assessing how conference reviewing has developed over the years, whether it's gotten better or actually worse. All right, so that was the experiment 2014. But by the way, the authors here have decided that the name change is retroactive. I never know. I never know when talking about old NeurIPS conferences, whether I'm supposed to say it was NIPS 2014 or NeurIPS. In any case, in this paper, we're doing NeurIPS. So what was the outcome of that experiment? And that's pretty interesting. Namely, here you can see these are still 2014 numbers, committee one and committee two split up. So it's not the same committee one, of course, but committee one would always be reviewers selected from kind of the first half of the population, committee two from the second half. They did agree on most of the papers, as you can see here. For 101 papers, they agreed to reject, for 22, they agreed to accept. However, for 43 of the papers, one committee would accept and the other one would actually reject. So for about 25% of the papers, the two committees would disagree. 25%, it's, you know, it sounds it's a lot, but it doesn't sound like that much. But if you look at it in a different way, where they say right here, if the conference reviewing had been run with a different committee, only half of the papers presented at the conference would have been the same. So this is looking at if you'd, for example, always go with committee one, you would have these papers. But if you would always go with committee two, you would have these papers. Therefore, but the simple selection of the committee determines about half the papers at the conference. So if you're at the conference, you walk through the big halls of posters, or you look at the proceedings, you have to keep in mind that half of the papers are there only purely because of the random choice of or not purely, but they wouldn't be there. Had the reviewing committee been a different one, half the papers, that's kind of crazy. And of course, this sparked a lot of discussion right here. So this is the outset, this was the results from that time. And now we're going into new analysis. So they do three different distinct points of analysis. The first one is they do the title is called reviewer calibration. So they try to figure out what portion of a reviewers assessment of a paper is, let's say objective, and what portion is subjective. So what portion of a score is simply due to the reviewers subjective feelings about the paper that doesn't match with any other reviewers scores. So here you can see this, for example, what you can do is you can build a model, you can build a model, you can say, why ij, that's the score that the jth reviewer gives to the ith paper. And you know, being the conference chairs, these authors here would have prime access to that data. So what you observe is why now you can say, we assume this is a combination of three things. First of all, we assume that there is some sort of a objective paper quality, which is fi. This is the objective quality of the paper. This is actually what the reviewers are trying to predict. So when the reviewer posts the number y into the system, they're trying their best to actually assess fi. However, there is also this bj right here. And this is the bias that the jth reviewer has in calibration. So not everyone, not everyone sees the one through 10 or one through nine scale that we have in the same fashion. And therefore, what's like a three to me might be a five to you. So we have to correct somehow for this and the inclusion of this bj factor is how we account for that. And then lastly, you have this eij factor right here. And this is the subjective portion of the score. So this is independent of the objective quality of the paper. This is sort of the subjective bonus or penalty that reviewer j gives to paper i. And our goal is going to be to figure out how do these two numbers compare to each other, how much of the score is objective versus subjective after we have calibrated for reviewer for general reviewer bias for calibration bias, let's say. Keep in mind, this is a model. This is how we imagine the world. All we observe is this y thing right here. What we can do is of course, we can put up a linear system of all the scores, right? And of all the scores, because every reviewer does give more than one score in this conference and every paper gets more than one reviewers scores. So we can put up a linear system. But it turns out this is over parameterized because you only have as many numbers as you have these parameters right here. So the rest both parameters, they don't, you don't have enough data points to assess that. Now as much fun as over parameterized models are in deep learning, they're actually not that good if you want to estimate a linear system. So what people do, they come up with regularizers and Bayesian approaches and yada, yada, yada. I'll skip all of this to just give you the numbers. So the model that these authors come up with determines that the factors of the linear systems are as follows. This here is the factor that goes with the fi. This one is the one that goes with the bj and this one is the one that goes with the ej. And you see you, you, you, you pull out this one and then you simply compare the number on the left to the number on the right and you'll see they're almost exactly the same. And that means and they formulate this here, in other words, 50% of a typical reviewer's score is coming from opinion that is particular to that reviewer and not shared with the other reviewers. This figure may seem large. Sorry about that. This figure may seem large, they say, but in retrospect, it's perhaps not surprising. So this is pretty, I guess this is pretty surprising to me, but it is not that, it is not that I didn't expect it. And I think anyone who's participated in conference peer review would expect a number that is in approximately this range because we know that the review process is pretty noisy. And very, very often individual reviewers just kind of give weird scores that you don't understand. And here's the reason you don't understand because it's the source of them are subjective and largely not shared by other reviewers. So having figured that out, having figured out that about 50% of the variation is due to just subjective feeling of a reviewer about a paper. Now they sort of try to validate their findings. And for that they run a simulation. So the simulation is, it's a simulated conference. So we assume that each paper was scored according to the model we've given above, and we estimated the accept consistency through averaging across 100,000 samples. So now they're simulating the conference with this experiment done. And they ask themselves, if this is really the correct model, then we should get back, we should get back a consistency of the 50% we found above. So because above the results of the experiments were that there was about a 50% consistency in acceptance in the experiment. And now they go and they look at all the papers and all the scores and they determine that there is about a 50% subjectivity in scoring. And now they ask themselves, do these two numbers match? And they run a simulation where every reviewer has a 50% subjectivity. And they ask themselves, if we simulate this splitting up into two committees, and then every committee agrees by themselves, do we see the numbers that we found in the experiment? And the answer is yes, actually. So you can see these are conferences for a bunch of different scenarios, namely for different number of reviewers, as you can see here, these are reviewers per committee. So random means there is no reviewer per committee, committee decisions are just random. And you can see that as the accept rate of the conference goes up, the accept precision of the committees go up because they simply they would more papers are accepted. And therefore, more papers would be the same if you were to change the committee. What we're interested in is, of course, the one with three reviewers, which is the most common reviewers scenario in these conferences. And that's this curve right here. So the way to read this is that, for example, if the conference had an accept rate of 50%, right here, then we would expect a reviewer consistency or an accept precision of 0.75 of 75%, which means that if we were to switch the reviewers for a particular or for all the papers, 75% of the paper would still be the same. Remember that in our experiment, only 50% of the papers were still the same if we switched committee. But the conference also didn't have a 50% accept rate. So for that, we actually need to go to the accept rate of the conference, which was something like 23% right here. And then if we look that up, we are at about a 60% accept precision. Now, this might still be away from the 50% we found in the experiment. However, the experiment had so little data that if you calculate the bounds on what the true accept precision was from that experiment, you can determine that it was between 38 and 64%. And the exact number we got is 61%. So this is still within the bounds of what we found in the experiment. So pretty interesting. This actually means that the model they put up is a close enough approximation to reality such that it predicts the experiment's outcome. And this gives us a little bit of a this gives us a little bit validation that we're on a good track right here. So we can sort of confidently say that about half of a reviewers decision on a particular paper essentially comes down to subjectivity is consistent with what we found in the experiment. And it'd be interesting to see how this develops this year when we repeat the experiment. So lastly, what they were trying to figure out is, well, are these reviews even worth it, so to say, do they actually predict how good a paper is, and you know, how do you measure how good a paper is, of course, by the number of citations. So here they define the citation impact as the log of the number of citations. And yes, there is a debate about whether citations really mean a paper is good or influential or blah, blah, blah. But we don't, for better or worse, we don't have a different measure right now than number of citations. And it's been seven years, which is like three generations in machine learning. So there is a long enough time that these papers had to accumulate citations. So let's just look at the accepted papers. Do the scores that the reviewers give to the papers predict in any way whether or not the paper is going to be cited more or less? So do higher scores indicate more citations? And the answer is no, not at all. So here is a plot. The correlation is 0.05. This is ever so slightly statistically significant, but not really. So you can, like at least for this particular conference right here, there's no correlation between reviewer scores and between reviewer scores and impact of the paper in the future. It becomes a little bit interesting when you ask specifically. So because here the question is, you know, is the paper novel? Is it correct? Is it well written and so on? These are not necessarily indicators of significance, right? If you accept the paper to a conference, only a small part of it is, is it significant? If you actually ask reviewers, do you think this paper will have a potentially major impact or not, you get a slightly higher correlation, but also not really, which means that reviewers are kind of bad at estimating whether any given paper will have a big impact or not. Though to be fair for most papers, the answer is probably no by default. However, the interesting part is when you ask them about their confidence in their rating and it is, if I understand correctly, it doesn't even matter which rating, but for the rating that you give at these conferences, you have to provide a confidence score. Like you say, okay, I think this paper is really good, but I'm not very confident. And if you simply correlate the confidence scores, as you can see here, the average confidence over all your sort of confidences of the paper with the impact, then you do get a slight correlation, which is interesting, right? So the authors here argue that it might be that there might be something like clarity in the paper. So if a paper is written very clearly, then you will also be able to understand it better as a reviewer, which makes your confidence higher. But also, since the paper is more clear, it means that the rest of the world will have an easier time understanding the paper and therefore cite it more often. So this is a good hypothesis, but it's quite interesting that the confidence in papers seems to predict the impact better than the actual assessment of the impact. That's astounding. It's not super astounding that confidence by itself would predict it, but that it does so more than if you directly ask people. I wonder what else we can ask. I wonder what weird questions we can ask that will then up correlating with the future impact. Do you like the colors of the paper? Do you like the pictures? So these were for accepted papers. They also interestingly trace the fate of the rejected papers. So they say only 414 were presented at the final conference. So they want to trace the rejected papers and they go through a lot of work to try to figure out where these papers ended up. So they search for papers with similar titles and authors or same titles and authors. And of course, this is not a perfect process, but it seems like they've been able to trace a lot of these papers to their final destination. You can see a lot of papers are discarded or some are simply posted on an archive or somewhere else. Of course, the discarded papers, you don't know if they somehow morphed into other papers or something like this. But it's still pretty interesting to see, though they say there are various error sources in these plots. Lastly, yeah, here is the fate of the rejected papers. Now they don't say exactly what blue and green means in this particular thing. In other plots in the same papers, they differentiate, for example, between papers that have been accepted somewhere else ultimately and papers that have not been or that they have not been able to trace. So this might be blue and green. I'm not sure. I haven't been able to maybe I'm just stupid at reading. But as you can see, if you look at the rejected papers, so this is the calibrated quality score for the rejected papers. And here you can see that there is in fact a correlation, which means that for the rejected papers, the assessment of the reviewers really does correlate with how the papers will end up doing ultimately. So I'm going to guess, well, if the citation count is in here, I'm going to guess the discarded paper must not be in here. Yeah, sorry. But the conclusion is that for the rejected papers, reviewers can tell whether they're better or worse. For the accepted papers, not so much. And that's what they said at the beginning. The review process is probably good at identifying bad papers, but bad at identifying good papers. And this is it's not too surprising because bad papers, you know, you can find it's really easy to recognize a very poor paper. But it's it's harder to recognize really how good a paper is, you know, compared to other good papers. So that was the paper they give some recommendations. For example, they say, well, maybe we should we should assess papers on on on some on different on different criteria than we do now. But they do guard they do warn against saying we should do away with with subjectivity all together. Because, you know, as annoying as the subjectivity is, they argue is it also guards against sort of the collective dominance. So it guards against sort of making consistent mistakes. If all the like if the entire conference for example, if the entire conference makes consistent mistakes in in some direction, then the subjectivity might counter that a little bit. I'm not sure if that's a super good argument. I am generally for noisy processes over super duper rigid ones. It seems though that the conference review right now is a bit too noisy. I'd rather do away with just having three reviewers and not having this accept barrier. This is my personal opinion. I would just do away with the accept barrier altogether. You know, you submit to a conference, you get a bunch of scores and then you have the scores. Like why do we need to divide papers up into accepted and rejected or, you know, like it seems better to just put papers out there and let the future let the future researchers assess them in retrospect, rather than having three random people with highly subjective opinions assess them. But yes, probably a bit of noise is good in a process like this. If you do a process like this. They also say, well, maybe we should not put that much value at publishing at top tier conferences. Now, I don't know how that's gonna work, you know, like whenever, whenever. And yeah, I wish I wish as well that we could like change the collective the collective thinking about our field. I don't I don't see that as a super easy task, though. In any case, this was the paper. Let me know your ideas. Let me know how you think this year's experiment is going to turn out. Like are we going to find more subjectivity? Are we going to find less? How much disagreement do you think we're going to find? This is going to be interesting. So yeah, thanks for listening and I'll see you next time.
 Hi there, today we'll look at inconsistency in conference peer review, revisiting the
 2014 NeurIPS experiment by Corina Cortes and Neil D. Lawrence, which were actually the
 chairs of the 2014 NeurIPS conference.
 So they are going to have access to some data that the rest of us sadly don't have access
 to.
 So it allows them to make pretty cool research on how conference reviewing works and whether
 or not it actually can determine the quality of a paper or how much of it is just random
 subjective reviewer decisions.
 Now this paper particularly here takes up the papers that were subject to the 2014 NeurIPS
 experiment and tracks them over time.
 So it looks at the papers that were submitted, how did they perform in the subsequent years,
 meaning how many citations that they accumulate, both for the accepted and for the rejected
 papers.
 And they find some pretty interesting results right here.
 So we'll dive into this.
 The paper is not too long and the conclusions are fairly straightforward.
 I still think it's really cool that people actually follow up on this work.
 So for those of you who don't know, the 2014 NeurIPS experiment, that is the wrong color,
 the 2014 NeurIPS experiment was an experiment in assessing how much of review of conference
 review is random essentially.
 So what you did was, and I think they have a little section about this here.
 So they selected about 10% of the submissions.
 These were 170 papers and these would undergo review by two separate committees.
 So whereas usually you have a paper that goes into a review, let's call that a committee,
 which is a bunch of reviewers and an area chair and they make the decisions of whether
 to accept or to reject.
 And yeah, at the end you have a decision.
 So in this experiment, you would take a paper, you would actually give it to two different
 committees, committee one and committee two.
 Committee one would only be selected from kind of one half of the reviewer pool and
 committee two would only be selected from the other half.
 These were random assignments to the two pools and also the papers who participated were
 randomly selected.
 So each of these committees would reach their own decision, accept or reject.
 And of course, the interesting part is how many of those agree or how many of those disagree
 with each other.
 And by the way, the paper would be accepted finally, if the max, so if either of the committees
 would accept the paper.
 And if I recall correctly, this year's NeurIPS conference actually repeats that experiment
 from 2014.
 So we're going to have another data point in hopefully assessing how conference reviewing
 has developed over the years, whether it's gotten better or actually worse.
 All right, so that was the experiment 2014.
 But by the way, the authors here have decided that the name change is retroactive.
 I never know.
 I never know when talking about old NeurIPS conferences, whether I'm supposed to say it
 was NIPS 2014 or NeurIPS.
 In any case, in this paper, we're doing NeurIPS.
 So what was the outcome of that experiment?
 And that's pretty interesting.
 Namely, here you can see these are still 2014 numbers, committee one and committee two split
 up.
 So it's not the same committee one, of course, but committee one would always be reviewers
 selected from kind of the first half of the population, committee two from the second
 half.
 They did agree on most of the papers, as you can see here.
 For 101 papers, they agreed to reject, for 22, they agreed to accept.
 However, for 43 of the papers, one committee would accept and the other one would actually
 reject.
 So for about 25% of the papers, the two committees would disagree.
 25%, it's, you know, it sounds it's a lot, but it doesn't sound like that much.
 But if you look at it in a different way, where they say right here, if the conference
 reviewing had been run with a different committee, only half of the papers presented at the conference
 would have been the same.
 So this is looking at if you'd, for example, always go with committee one, you would have
 these papers.
 But if you would always go with committee two, you would have these papers.
 Therefore, but the simple selection of the committee determines about half the papers
 at the conference.
 So if you're at the conference, you walk through the big halls of posters, or you look at the
 proceedings, you have to keep in mind that half of the papers are there only purely because
 of the random choice of or not purely, but they wouldn't be there.
 Had the reviewing committee been a different one, half the papers, that's kind of crazy.
 And of course, this sparked a lot of discussion right here.
 So this is the outset, this was the results from that time.
 And now we're going into new analysis.
 So they do three different distinct points of analysis.
 The first one is they do the title is called reviewer calibration.
 So they try to figure out what portion of a reviewers assessment of a paper is, let's
 say objective, and what portion is subjective.
 So what portion of a score is simply due to the reviewers subjective feelings about the
 paper that doesn't match with any other reviewers scores.
 So here you can see this, for example, what you can do is you can build a model, you can
 build a model, you can say, why ij, that's the score that the jth reviewer gives to the
 ith paper.
 And you know, being the conference chairs, these authors here would have prime access
 to that data.
 So what you observe is why now you can say, we assume this is a combination of three things.
 First of all, we assume that there is some sort of a objective paper quality, which is
 fi.
 This is the objective quality of the paper.
 This is actually what the reviewers are trying to predict.
 So when the reviewer posts the number y into the system, they're trying their best to actually
 assess fi.
 However, there is also this bj right here.
 And this is the bias that the jth reviewer has in calibration.
 So not everyone, not everyone sees the one through 10 or one through nine scale that
 we have in the same fashion.
 And therefore, what's like a three to me might be a five to you.
 So we have to correct somehow for this and the inclusion of this bj factor is how we
 account for that.
 And then lastly, you have this eij factor right here.
 And this is the subjective portion of the score.
 So this is independent of the objective quality of the paper.
 This is sort of the subjective bonus or penalty that reviewer j gives to paper i.
 And our goal is going to be to figure out how do these two numbers compare to each other,
 how much of the score is objective versus subjective after we have calibrated for reviewer
 for general reviewer bias for calibration bias, let's say.
 Keep in mind, this is a model.
 This is how we imagine the world.
 All we observe is this y thing right here.
 What we can do is of course, we can put up a linear system of all the scores, right?
 And of all the scores, because every reviewer does give more than one score in this conference
 and every paper gets more than one reviewers scores.
 So we can put up a linear system.
 But it turns out this is over parameterized because you only have as many numbers as you
 have these parameters right here.
 So the rest both parameters, they don't, you don't have enough data points to assess that.
 Now as much fun as over parameterized models are in deep learning, they're actually not
 that good if you want to estimate a linear system.
 So what people do, they come up with regularizers and Bayesian approaches and yada, yada, yada.
 I'll skip all of this to just give you the numbers.
 So the model that these authors come up with determines that the factors of the linear
 systems are as follows.
 This here is the factor that goes with the fi.
 This one is the one that goes with the bj and this one is the one that goes with the
 ej.
 And you see you, you, you, you pull out this one and then you simply compare the number
 on the left to the number on the right and you'll see they're almost exactly the same.
 And that means and they formulate this here, in other words, 50% of a typical reviewer's
 score is coming from opinion that is particular to that reviewer and not shared with the other
 reviewers.
 This figure may seem large.
 Sorry about that.
 This figure may seem large, they say, but in retrospect, it's perhaps not surprising.
 So this is pretty, I guess this is pretty surprising to me, but it is not that, it is
 not that I didn't expect it.
 And I think anyone who's participated in conference peer review would expect a number that is
 in approximately this range because we know that the review process is pretty noisy.
 And very, very often individual reviewers just kind of give weird scores that you don't understand.
 And here's the reason you don't understand because it's the source of them are subjective
 and largely not shared by other reviewers.
 So having figured that out, having figured out that about 50% of the variation is due
 to just subjective feeling of a reviewer about a paper.
 Now they sort of try to validate their findings.
 And for that they run a simulation.
 So the simulation is, it's a simulated conference.
 So we assume that each paper was scored according to the model we've given above, and we estimated
 the accept consistency through averaging across 100,000 samples.
 So now they're simulating the conference with this experiment done.
 And they ask themselves, if this is really the correct model, then we should get back,
 we should get back a consistency of the 50% we found above.
 So because above the results of the experiments were that there was about a 50% consistency
 in acceptance in the experiment.
 And now they go and they look at all the papers and all the scores and they determine that
 there is about a 50% subjectivity in scoring.
 And now they ask themselves, do these two numbers match?
 And they run a simulation where every reviewer has a 50% subjectivity.
 And they ask themselves, if we simulate this splitting up into two committees, and then
 every committee agrees by themselves, do we see the numbers that we found in the experiment?
 And the answer is yes, actually.
 So you can see these are conferences for a bunch of different scenarios, namely for different
 number of reviewers, as you can see here, these are reviewers per committee.
 So random means there is no reviewer per committee, committee decisions are just random.
 And you can see that as the accept rate of the conference goes up, the accept precision
 of the committees go up because they simply they would more papers are accepted.
 And therefore, more papers would be the same if you were to change the committee.
 What we're interested in is, of course, the one with three reviewers, which is the most
 common reviewers scenario in these conferences.
 And that's this curve right here.
 So the way to read this is that, for example, if the conference had an accept rate of 50%,
 right here, then we would expect a reviewer consistency or an accept precision of 0.75
 of 75%, which means that if we were to switch the reviewers for a particular or for all
 the papers, 75% of the paper would still be the same.
 Remember that in our experiment, only 50% of the papers were still the same if we switched
 committee.
 But the conference also didn't have a 50% accept rate.
 So for that, we actually need to go to the accept rate of the conference, which was something
 like 23% right here.
 And then if we look that up, we are at about a 60% accept precision.
 Now, this might still be away from the 50% we found in the experiment.
 However, the experiment had so little data that if you calculate the bounds on what the
 true accept precision was from that experiment, you can determine that it was between 38 and
 64%.
 And the exact number we got is 61%.
 So this is still within the bounds of what we found in the experiment.
 So pretty interesting.
 This actually means that the model they put up is a close enough approximation to reality
 such that it predicts the experiment's outcome.
 And this gives us a little bit of a this gives us a little bit validation that we're on a
 good track right here.
 So we can sort of confidently say that about half of a reviewers decision on a particular
 paper essentially comes down to subjectivity is consistent with what we found in the experiment.
 And it'd be interesting to see how this develops this year when we repeat the experiment.
 So lastly, what they were trying to figure out is, well, are these reviews even worth
 it, so to say, do they actually predict how good a paper is, and you know, how do you
 measure how good a paper is, of course, by the number of citations.
 So here they define the citation impact as the log of the number of citations.
 And yes, there is a debate about whether citations really mean a paper is good or influential
 or blah, blah, blah.
 But we don't, for better or worse, we don't have a different measure right now than number
 of citations.
 And it's been seven years, which is like three generations in machine learning.
 So there is a long enough time that these papers had to accumulate citations.
 So let's just look at the accepted papers.
 Do the scores that the reviewers give to the papers predict in any way whether or not the
 paper is going to be cited more or less?
 So do higher scores indicate more citations?
 And the answer is no, not at all.
 So here is a plot.
 The correlation is 0.05.
 This is ever so slightly statistically significant, but not really.
 So you can, like at least for this particular conference right here, there's no correlation
 between reviewer scores and between reviewer scores and impact of the paper in the future.
 It becomes a little bit interesting when you ask specifically.
 So because here the question is, you know, is the paper novel?
 Is it correct?
 Is it well written and so on?
 These are not necessarily indicators of significance, right?
 If you accept the paper to a conference, only a small part of it is, is it significant?
 If you actually ask reviewers, do you think this paper will have a potentially major impact
 or not, you get a slightly higher correlation, but also not really, which means that reviewers
 are kind of bad at estimating whether any given paper will have a big impact or not.
 Though to be fair for most papers, the answer is probably no by default.
 However, the interesting part is when you ask them about their confidence in their rating
 and it is, if I understand correctly, it doesn't even matter which rating, but for the rating
 that you give at these conferences, you have to provide a confidence score.
 Like you say, okay, I think this paper is really good, but I'm not very confident.
 And if you simply correlate the confidence scores, as you can see here, the average confidence
 over all your sort of confidences of the paper with the impact, then you do get a slight
 correlation, which is interesting, right?
 So the authors here argue that it might be that there might be something like clarity
 in the paper.
 So if a paper is written very clearly, then you will also be able to understand it better
 as a reviewer, which makes your confidence higher.
 But also, since the paper is more clear, it means that the rest of the world will have
 an easier time understanding the paper and therefore cite it more often.
 So this is a good hypothesis, but it's quite interesting that the confidence in papers
 seems to predict the impact better than the actual assessment of the impact.
 That's astounding.
 It's not super astounding that confidence by itself would predict it, but that it does
 so more than if you directly ask people.
 I wonder what else we can ask.
 I wonder what weird questions we can ask that will then up correlating with the future impact.
 Do you like the colors of the paper?
 Do you like the pictures?
 So these were for accepted papers.
 They also interestingly trace the fate of the rejected papers.
 So they say only 414 were presented at the final conference.
 So they want to trace the rejected papers and they go through a lot of work to try to
 figure out where these papers ended up.
 So they search for papers with similar titles and authors or same titles and authors.
 And of course, this is not a perfect process, but it seems like they've been able to trace
 a lot of these papers to their final destination.
 You can see a lot of papers are discarded or some are simply posted on an archive or
 somewhere else.
 Of course, the discarded papers, you don't know if they somehow morphed into other papers
 or something like this.
 But it's still pretty interesting to see, though they say there are various error sources
 in these plots.
 Lastly, yeah, here is the fate of the rejected papers.
 Now they don't say exactly what blue and green means in this particular thing.
 In other plots in the same papers, they differentiate, for example, between papers that have been
 accepted somewhere else ultimately and papers that have not been or that they have not been
 able to trace.
 So this might be blue and green.
 I'm not sure.
 I haven't been able to maybe I'm just stupid at reading.
 But as you can see, if you look at the rejected papers, so this is the calibrated quality
 score for the rejected papers.
 And here you can see that there is in fact a correlation, which means that for the rejected
 papers, the assessment of the reviewers really does correlate with how the papers will end
 up doing ultimately.
 So I'm going to guess, well, if the citation count is in here, I'm going to guess the discarded
 paper must not be in here.
 Yeah, sorry.
 But the conclusion is that for the rejected papers, reviewers can tell whether they're
 better or worse.
 For the accepted papers, not so much.
 And that's what they said at the beginning.
 The review process is probably good at identifying bad papers, but bad at identifying good papers.
 And this is it's not too surprising because bad papers, you know, you can find it's really
 easy to recognize a very poor paper.
 But it's it's harder to recognize really how good a paper is, you know, compared to other
 good papers.
 So that was the paper they give some recommendations.
 For example, they say, well, maybe we should we should assess papers on on on some on different
 on different criteria than we do now.
 But they do guard they do warn against saying we should do away with with subjectivity all
 together.
 Because, you know, as annoying as the subjectivity is, they argue is it also guards against sort
 of the collective dominance.
 So it guards against sort of making consistent mistakes.
 If all the like if the entire conference for example, if the entire conference makes consistent
 mistakes in in some direction, then the subjectivity might counter that a little bit.
 I'm not sure if that's a super good argument.
 I am generally for noisy processes over super duper rigid ones.
 It seems though that the conference review right now is a bit too noisy.
 I'd rather do away with just having three reviewers and not having this accept barrier.
 This is my personal opinion.
 I would just do away with the accept barrier altogether.
 You know, you submit to a conference, you get a bunch of scores and then you have the
 scores.
 Like why do we need to divide papers up into accepted and rejected or, you know, like it
 seems better to just put papers out there and let the future let the future researchers
 assess them in retrospect, rather than having three random people with highly subjective
 opinions assess them.
 But yes, probably a bit of noise is good in a process like this.
 If you do a process like this.
 They also say, well, maybe we should not put that much value at publishing at top tier
 conferences.
 Now, I don't know how that's gonna work, you know, like whenever, whenever.
 And yeah, I wish I wish as well that we could like change the collective the collective thinking
 about our field.
 I don't I don't see that as a super easy task, though.
 In any case, this was the paper.
 Let me know your ideas.
 Let me know how you think this year's experiment is going to turn out.
 Like are we going to find more subjectivity?
 Are we going to find less?
 How much disagreement do you think we're going to find?
 This is going to be interesting.
 So yeah, thanks for listening and I'll see you next time.