[News] The NeurIPS Broader Impact Statement
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper neurips conference ethics society impact statement submission authors accept reject flag review double blind
For the first time, all authors submitting to the NeurIPS conference are forced to write a statement about the broader impact of their research on society. The messaging around this and how exactly this can influence the paper acceptance process is highly confusing.

OUTLINE:
0:00 - Intro
0:30 - VentureBeat Article
1:35 - Official Communication
9:55 - Special Ethics Reviewers
11:00 - Unofficial Communication
22:55 - Conclusion

Sources:
https://neurips.cc/Conferences/2020/CallForPapers
https://neurips.cc/Conferences/2020/PaperInformation/ReviewerGuidelines
https://neurips.cc/Conferences/2020/PaperInformation/NeurIPS-FAQ
https://medium.com/@NeurIPSConf/getting-started-with-neurips-2020-e350f9b39c28
https://venturebeat.com/2020/02/24/neurips-requires-ai-researchers-to-account-for-societal-impact-and-financial-conflicts-of-interest/
https://medium.com/@NeurIPSConf/a-note-for-submitting-authors-48cebfebae82
https://medium.com/@BrentH/suggestions-for-writing-neurips-2020-broader-impacts-statements-121da1b765bf
https://acm-fca.org/2018/03/29/negativeimpacts/
https://medium.com/@operations_18894/a-guide-to-writing-the-neurips-impact-statement-4293b723f832
https://gdpr-info.eu/

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 As many of you might be familiar with, the NURIPS 2020 conference now requires authors to include a section in their submissions discussing the broader impact of their work, including possible societal consequences, both positive and negative. That was announced in the Getting Started with NURIPS 2020 announcement on Medium by the conference organizers. Shortly after that, in an email to VentureBeat, Michael Lippman, the communications chair of NURIPS 2020, told VentureBeat that these statements will be published with each paper. However, they'll appear only in the camera ready versions of the papers, so they do not compromise the double blind nature of the reviewing process. But then goes on to say, reviewers and area chairs assessment will be done on the basis of technical contributions only. However, if a paper is flagged for potential ethical concerns, then the paper will be sent to another set of reviewers with expertise in ethics and machine learning. The final acceptance of these papers is contingent on the positive assessment by these second set of reviewers as well. So this seems a bit odd. For one, the broader impact statement is only published after the double blind reviewing process is over. But the papers will be assessed based on their ethical and societal impact. So maybe the assessment will have nothing to do with this statement. Let's dive in a bit deeper. In the NURIPS 2020 FAQ, do I have to complete the broader impact section? The answer is yes. Please include the section. But they say, however, if your work is very theoretical or is general enough that there is no particular application foreseen, then you are free to write a broader impact discussion is not applicable. So until now, I genuinely feel that the conference organizers view this as some sort of experiment. And it is reasonable if it doesn't apply to you, which is probably the case for most people, then you can simply write, this does not apply to our research. Can my submission be rejected solely on the basis of the broader impact section? Answer, no. Reviewers will be asked to rate a submission based on the evaluation criteria, not your broader impact section. They will also be asked to check whether the broader impact section is adequately addressed. So reviewers will be able to check the broader impact section, which isn't there or is it there during the double blind reviewing process. But they only have to say whether it's adequately addressed and they will not be able to reject a paper on that basis. Again, they repeat. The authors can simply state this work does not present any foreseeable societal consequences if the authors feel that this is the case. If this is not the case, the conference asks of the authors to discuss along the lines of positive potential impacts and negative potential impacts of the submission. So far, so good. Let's actually look at these evaluation criteria that they ask reviewers to grade the paper by, which, as they say, has nothing to do with the broader impact section. Papers that violate the style have already been published or have fatal flaws may be rejected on that basis. Other submissions will be judged on the basis of their technical quality, novelty, potential impact and clarity. But one could still think that the potential impact here is a potential technical impact. It has nothing to do with this broader impact section. That has nothing to do with you being accepted or rejected. They go on to say submissions will also be considered on ethical grounds, regardless of scientific quality or contribution. And they say a submission may be rejected for ethical considerations. Now, again, one could say that they don't look at your broader impact statement if they feel that there is an ethical violation, they reject it. But before we've already heard that if the reviewers feel that there is an ethical consideration that may include the broader impact section, they can flag the paper and that will go to a set of second reviewers. And these reviewers can actually reject your paper. So it seems like there is a bit of a mixed message here. The entire question sort of hinges on who makes the decision and based on what. And one of the questions is what kind of decisions do the reviewers make? So where else better to go than the reviewer guidelines? Question 11 to the reviewers. Have the authors adequately address the broader impact of their work, including potential negative ethical and societal implications of their work? And indicate whether you believe the broader impact section was adequate. So it feels like that the reviewers are simply to evaluate whether this has done with enough work and not necessarily whether they agree with the broader impact section or not. The question here is if the reviewers think that this has not been done with enough adequacy, but don't necessarily see an ethical problem or actually do, can it be rejected on the basis that it has not been done adequately? The entire writing here seems like it should. But then also it seems like the reviewers assessment should have nothing to do with the broader impact section. Question 12 of the reviewer guidelines says, does the submission raise potential ethical concerns? Note that this is a different question from question 11, where you're simply asked to judge adequacy. The reviewer guidelines say, note that your rating should be independent of this. If the AC also shares this concern, dedicated reviewers with expertise at the intersection of ethics and machine learning will further review the submission. And your duty is to flag the papers. This now seems that reviewers are to consider the adequacy of the statement, but not its content and forward its content to another section, which contradicts that the reviewers don't see the statement or that the statement can't influence the review. And it also contradicts the statement that your paper cannot be rejected based on the broader impact section. Namely, if the second set of reviewers read your broader impact section, find it doesn't address their concerns, they can in fact reject your paper based on that. I guess someone arguing against that would say that these people could also reject it just because they think it's ethically problematic. But if the paper has a broader impact section, I think they are going to look at that with some sort of an open mind and at least be influenced by that. In a note for submitting authors, the conference organizers again released a statement saying that the broader impact statement should include a statement about the foreseeable positive impact as well as potential risks associated mitigations of the proposed research. Authors can also declare that a broader impact statement is not applicable to their work if they believe this to be the case. And they again repeat reviewers will also confirm whether the broader impact statement is inadequate. But this assessment will not affect the overall rating. However, reviewers have the option to flag a paper for ethical concerns, which may relate to the content of the broader impact section. The paper will be sent for additional review to a pool of emergency reviewers with expertise in machine learning and ethics who will provide an assessment solely on the basis of ethical considerations. We expect very few, if any, papers to need such further assessment. So the official communication makes a divide between on one hand, adequacy and on the other hand, real ethical concerns. And their message is basically reviewers will judge the adequacy, flag the ethical concerns, and then special reviewers will be able to reject based on ethical concerns. Now, what's not really clear is the messaging that reviewers should not base their judgment on the broader impact section. But then where does this adequacy rating go into the process of rejecting or accepting a paper? And with them saying there will only be a few, they expect only a few, it seems like it's sort of an experiment that they do this year. Oh, hey, NURIBS organizer. Well, hello. So you've decided to make everyone include a broader impact statement, but the broader impact statement will only be visible after the reviewing process. Correct. But the reviewers should check its adequacy during the review process. Yes, while they can't see it. Correct. But it should in no way influence their judgment and you can't be rejected because of that. That is correct. But if it is found to be inadequate or problematic, it is sent to a second set of reviewers, which on the basis of the paper and the broader impact statement will decide if the paper is of ethical concern. Yes. And if the paper is of enough ethical concern and the broader impact statement doesn't convince the special reviewers otherwise, they will be able to reject that paper. That is indeed correct. So how are you saying that the broader impact statement has no influence on your score and you can't be rejected because of it? Well, as we said, no one's able to see it until the paper is released. Obviously. Let's talk about these special reviewers. And for that, we broadly have to talk about incentives. Just imagine for a second that this expectation comes to fruit, that no paper is actually flagged and or any paper that is flagged to this committee will come back with a clear, no, this is not really an ethical concern or a reason to discard this paper as a scientific contribution. One might almost think that then this program will be abolished in the next year because it's useless. So the more problems the special reviewers find, the more justified their position is. I wonder where that leads. I guess we are very dependent on pretty much every single person in these special reviewers being some sort of super honest person that has no incentives and also no strong opinions on these things and and generally has gone into this ML ethics just out of interest and not to actually make an impact. I'm sure that will work out just fine. Now, the official NURiBS website actually links to a blog post of Brent Hecht, suggestions for writing the NURiBS 2020 broader impact statements. So we can reasonably assume that this is at least in agreement with the organizers of the conference. Brent here says understanding the societal impacts of your work is going to be hard. It is going to take lots of effort to write NURiBS broader impact statements. Tons of work has already been done for you. Check out the literature from communities that have studied societal impacts of AI for a long while. Even better, bring a social scientist onto your research team. Remember, though, they don't work for free. Hire them into your company, give them subawards, recruit them as PhD students through interdisciplinary programs. So are you saying the more problems these people find, the more of them will get hired? And again, look at these statements in general. It seems to really be about how much work are you able to put into this? So here's your average PhD student. Now, they pretty much already have to write their papers by themselves or in very, very small teams because they need first authorship. And they have to do all their experiments and they don't have enough resources. And now they're also asked to spend considerable amount of time not only writing this very hard statement, but also reading up on all the literature that there is to read up on. Or alternatively, if you don't want to do that, well, just hire someone, of course, because budgets and salaries in universities and for PhD students is notoriously loose. We can just hire someone that does that. I don't really think it's possible for single PhD students or research labs at universities to just hire someone or put someone full time on this additional required work that is put onto them. I wonder who will be actually able to put additional people on this such that they surely end up with beautiful, well-researched, broader impact statements to justify even the most ethically concerning research. I'm wondering, it just gets on the tip of my tongue, but I was going to have to leave that for another time. For people who do more theoretical work, it is going to be more difficult. Wait a minute. I thought the official communication was you're very free to leave it away if you don't think this applies to you. But here it basically saying it's going to be more work for you. Find something that is both rigorous and practical for your research. And the argument is to get funding for any theoretical work, someone had to make an argument about positive societal impacts at some point. Not true. Some universities simply get money and academic freedom. If that argument is possible, it is probably also possible to make a rigorous statement about some negative societal impacts. You might be tempted to write boilerplate low information statements. Don't do this. It will undermine the rigor in the rest of your paper. The public will roll its eyes and reviewers may and often should call you out. Now, wait a minute. I thought the reviewers are only supposed to judge the adequacy and absolutely have no influence on their judgment of the paper. But the underpinning of this text here is basically that if you as a reviewer feel that this hasn't been done adequately, you sort of should let this swap over into your assessment of the rigor and adequacy of the technical contributions in the paper. Because they're kind of the same, right? And they also say that this might spark a conversation is specifically the author response period will be a decent opportunity to have a bit of a dialogue between author and reviewer on the impact statement. So this basically means that I as a reviewer now have to write in my review something about the broader impact statement and not only judge its adequacy in a special field for it. And then the author is forced to spend a bunch of their very, very, very valuable author response on rebuttling the reviewers assessment of their broader impact statement. Our proposal's view is that it's not your job as a reviewer to judge submissions for their impact. Rather, you should evaluate the rigor with which they disclose their impacts. So again, it's about putting work into it. If you go to the full proposal behind this blog post, you'll find the following snippets. So there is a list of expected outcomes of introducing such mandatory broader impact statements. Now, have a look at this expected outcomes. We expect that action on the above recommendations will lead to a number of desirable outcomes. And they're all positive outcomes. Now, haven't we been discussing for the last minutes that it's always important to assess the positive and the negative outcomes of your actions and of your releases? How ironic that none of the organizers of the conference, nor any of these people communicating, were forced to release a broader impact statement discussing the negative consequences that their actions would have on the community and the greater society. They go on to give a list of examples of how you could do such positive and negative aspects of technology. One of them is social media. And I would agree that there are ethical considerations if you invent social media. We all know that social media can be some sort of a dopamine feedback loop addiction and have negative consequences in society that are not readily visible. But it goes on. Crowdwork, a researcher who invents a new crowdwork framework, likely motivates her work by highlighting the problem the framework solves. But they go on to say that crowdwork also has negative externalities, such as incentivizing very low pay. And the researcher should find ways to engineer her crowdwork framework such that these externalities are structurally mitigated and or she might advocate for minimum wage laws to be adapted. So this researcher working out a problem in crowdworking now has to basically solve millennia old problems in structural economics that have thousands of moving parts and no clear consensus on how to solve. But the best example, and this is the example they actually tell you to look at if your work is more theoretical or you don't really think it has an impact, is the following storage and computation. Recent advances in storage systems and graphical processing unit processing afford the easy storage of massive amounts of data and the real time computation on these data. This has incentivized corporations to collect every possible data point about their users, save this data indefinitely and strive to monetize this data in new ways. While allowing for impressive new capabilities, this trend also presents tremendous risks to privacy. Researchers working in storage and GPU processing should consider these and other foreseeable potential risks in their papers. They should also enumerate technological and policy means by which these risks might be mitigated, e.g. technologies to automate general data protection regulation require capabilities and improvements to GDPR like policies. That is absolutely mad. So here you are making a GPU chip more powerful and you're asked to think ahead about the fact that this can be used to mine data. And not only that, now you're also required to propose improvements to GDPR like policies. The GDPR only an 88 page, very fine print legal document that in addition to all the literature about AI governance, our poor PhD student is now also required to read, understand and be able to improve. How long does this chain of causality go? How do you have to think ahead? This gets ridiculous. It's 200,000 BC and Nuno in his cave just invented fire. Well, fire can be used to cook food, can be used to have less disease, can be used to settle down, expand civilization, build educational facilities, build up a culture, a scientific method, enable massive progress, industrialization, general improvement in health, wealth, education and happiness of society, which ultimately leads to some people building GPUs and saving your data and analyzing all your things to serve you ads of better kitty stickers. How could Nuno in his cave do this to us? Where is his broader impact statement about the invention of fire for the future data collection algorithms on GPUs? Look, I'm not saying that you should not consider the downstreams effect of your inventions. Of course you should. But at some point it gets ridiculous for most of the work handed into a conference like NURIPS. Either the downstreams effect are so far away that is almost impossible to foresee or as any technology you can use it for good and for bad. And it is going to be with the application of this technology and not its invention where the good and the bad come in. And what most people are going to do is simply come up with things that mean absolutely nothing and generally make not a lot of difference. While it gives a big advantage to big institutions that can spend a lot of time and effort on crafting very rigorous adequate statements. Another release called A Guide to Writing the NURIPS Impact Statement that is not linked by NURIPS, but as they say was in communication with some of the organizers of the conference. So it's reasonable to assume they also largely agree with these positions here. Says you should discuss, read and reflect time permitting impact assessment will benefit from broad intellectual reflection, discuss potential impacts, follow public discussion, read case studies and read the scholarly literature. On tech governance, of course, time permitting. But then again, if it's not rigorous enough, a reviewer might be getting the idea that the rest of your paper isn't rigorous enough. So maybe time must permit for this one. And they again say, think about impacts even for theoretical work. So the official communication always says that if you don't feel this applies to you, you're very free to write. This doesn't apply to me. But the in official communication says if this doesn't apply, you're doing something wrong. And by the way, we're evaluate the rest of your paper based on the amount of work you put into that statement. Ultimately, these statements are just going to boil down to you can do good and bad things with any technology as is visible on this example they give here. Pluribus, a superhuman AI for multiplayer poker, they say they intentionally choose to broaden the focus of their broader impact assessment, depending who can use this scientific advance such as criminals or well motivated citizens. This technology may be socially harmful or beneficial. If access to this capability is mostly available to the wealthy, it could plausibly promote concentration of wealth. And further on the other side, increased skill could increase total welfare. Gee, if that doesn't apply to every single technology ever, I don't know. Again, my general assessment of this is not that it is absolutely wrong to do this or very useless. It is just shifting the balance a bit more on to large institutions who can actually afford to spend a lot of time and work into crafting beautiful statements. And in general, I don't think it's that big of a deal, but I also don't think it's going to help very much to just force everyone to do this. I guess we'll see how it turns out. In this VentureBeat article, they link someone named Joe Redmond saying, I stopped doing CV research because I saw the impact my work was having. I love the work, but the military applications and privacy concerns eventually became impossible to ignore, which I respect a lot. But I would ask, did Joe Redmond realize this after being forced to write a broader impact statement or at some other point? That was my two cents. If you like videos like this and paper analysis and other things, then subscribe, like wherever these buttons are, share it with your friends and see you next time.
 As many of you might be familiar with, the NURIPS 2020 conference now requires authors to include a section in their submissions discussing the broader impact of their work, including possible societal consequences, both positive and negative.
 That was announced in the Getting Started with NURIPS 2020 announcement on Medium by the conference organizers.
 Shortly after that, in an email to VentureBeat, Michael Lippman, the communications chair of NURIPS 2020, told VentureBeat that these statements will be published with each paper.
 However, they'll appear only in the camera ready versions of the papers, so they do not compromise the double blind nature of the reviewing process.
 But then goes on to say, reviewers and area chairs assessment will be done on the basis of technical contributions only.
 However, if a paper is flagged for potential ethical concerns, then the paper will be sent to another set of reviewers with expertise in ethics and machine learning.
 The final acceptance of these papers is contingent on the positive assessment by these second set of reviewers as well.
 So this seems a bit odd. For one, the broader impact statement is only published after the double blind reviewing process is over.
 But the papers will be assessed based on their ethical and societal impact.
 So maybe the assessment will have nothing to do with this statement. Let's dive in a bit deeper.
 In the NURIPS 2020 FAQ, do I have to complete the broader impact section?
 The answer is yes. Please include the section.
 But they say, however, if your work is very theoretical or is general enough that there is no particular application foreseen, then you are free to write a broader impact discussion is not applicable.
 So until now, I genuinely feel that the conference organizers view this as some sort of experiment.
 And it is reasonable if it doesn't apply to you, which is probably the case for most people, then you can simply write, this does not apply to our research.
 Can my submission be rejected solely on the basis of the broader impact section? Answer, no.
 Reviewers will be asked to rate a submission based on the evaluation criteria, not your broader impact section.
 They will also be asked to check whether the broader impact section is adequately addressed.
 So reviewers will be able to check the broader impact section, which isn't there or is it there during the double blind reviewing process.
 But they only have to say whether it's adequately addressed and they will not be able to reject a paper on that basis.
 Again, they repeat. The authors can simply state this work does not present any foreseeable societal consequences if the authors feel that this is the case.
 If this is not the case, the conference asks of the authors to discuss along the lines of positive potential impacts and negative potential impacts of the submission.
 So far, so good. Let's actually look at these evaluation criteria that they ask reviewers to grade the paper by, which, as they say, has nothing to do with the broader impact section.
 Papers that violate the style have already been published or have fatal flaws may be rejected on that basis.
 Other submissions will be judged on the basis of their technical quality, novelty, potential impact and clarity.
 But one could still think that the potential impact here is a potential technical impact.
 It has nothing to do with this broader impact section. That has nothing to do with you being accepted or rejected.
 They go on to say submissions will also be considered on ethical grounds, regardless of scientific quality or contribution.
 And they say a submission may be rejected for ethical considerations.
 Now, again, one could say that they don't look at your broader impact statement if they feel that there is an ethical violation, they reject it.
 But before we've already heard that if the reviewers feel that there is an ethical consideration that may include the broader impact section, they can flag the paper and that will go to a set of second reviewers.
 And these reviewers can actually reject your paper. So it seems like there is a bit of a mixed message here.
 The entire question sort of hinges on who makes the decision and based on what.
 And one of the questions is what kind of decisions do the reviewers make?
 So where else better to go than the reviewer guidelines?
 Question 11 to the reviewers.
 Have the authors adequately address the broader impact of their work, including potential negative ethical and societal implications of their work?
 And indicate whether you believe the broader impact section was adequate.
 So it feels like that the reviewers are simply to evaluate whether this has done with enough work and not necessarily whether they agree with the broader impact section or not.
 The question here is if the reviewers think that this has not been done with enough adequacy, but don't necessarily see an ethical problem or actually do, can it be rejected on the basis that it has not been done adequately?
 The entire writing here seems like it should.
 But then also it seems like the reviewers assessment should have nothing to do with the broader impact section.
 Question 12 of the reviewer guidelines says, does the submission raise potential ethical concerns?
 Note that this is a different question from question 11, where you're simply asked to judge adequacy.
 The reviewer guidelines say, note that your rating should be independent of this.
 If the AC also shares this concern, dedicated reviewers with expertise at the intersection of ethics and machine learning will further review the submission.
 And your duty is to flag the papers.
 This now seems that reviewers are to consider the adequacy of the statement, but not its content and forward its content to another section, which contradicts that the reviewers don't see the statement or that the statement can't influence the review.
 And it also contradicts the statement that your paper cannot be rejected based on the broader impact section.
 Namely, if the second set of reviewers read your broader impact section, find it doesn't address their concerns, they can in fact reject your paper based on that.
 I guess someone arguing against that would say that these people could also reject it just because they think it's ethically problematic.
 But if the paper has a broader impact section, I think they are going to look at that with some sort of an open mind and at least be influenced by that.
 In a note for submitting authors, the conference organizers again released a statement saying that the broader impact statement should include a statement about the foreseeable positive impact as well as potential risks associated mitigations of the proposed research.
 Authors can also declare that a broader impact statement is not applicable to their work if they believe this to be the case.
 And they again repeat reviewers will also confirm whether the broader impact statement is inadequate.
 But this assessment will not affect the overall rating.
 However, reviewers have the option to flag a paper for ethical concerns, which may relate to the content of the broader impact section.
 The paper will be sent for additional review to a pool of emergency reviewers with expertise in machine learning and ethics who will provide an assessment solely on the basis of ethical considerations.
 We expect very few, if any, papers to need such further assessment.
 So the official communication makes a divide between on one hand, adequacy and on the other hand, real ethical concerns.
 And their message is basically reviewers will judge the adequacy, flag the ethical concerns, and then special reviewers will be able to reject based on ethical concerns.
 Now, what's not really clear is the messaging that reviewers should not base their judgment on the broader impact section.
 But then where does this adequacy rating go into the process of rejecting or accepting a paper?
 And with them saying there will only be a few, they expect only a few, it seems like it's sort of an experiment that they do this year.
 Oh, hey, NURIBS organizer. Well, hello.
 So you've decided to make everyone include a broader impact statement, but the broader impact statement will only be visible after the reviewing process.
 Correct. But the reviewers should check its adequacy during the review process.
 Yes, while they can't see it.
 Correct. But it should in no way influence their judgment and you can't be rejected because of that.
 That is correct. But if it is found to be inadequate or problematic, it is sent to a second set of reviewers,
 which on the basis of the paper and the broader impact statement will decide if the paper is of ethical concern.
 Yes. And if the paper is of enough ethical concern and the broader impact statement doesn't convince the special reviewers otherwise,
 they will be able to reject that paper. That is indeed correct.
 So how are you saying that the broader impact statement has no influence on your score and you can't be rejected because of it?
 Well, as we said, no one's able to see it until the paper is released. Obviously.
 Let's talk about these special reviewers. And for that, we broadly have to talk about incentives.
 Just imagine for a second that this expectation comes to fruit, that no paper is actually flagged and or any paper that is flagged to this committee
 will come back with a clear, no, this is not really an ethical concern or a reason to discard this paper as a scientific contribution.
 One might almost think that then this program will be abolished in the next year because it's useless.
 So the more problems the special reviewers find, the more justified their position is.
 I wonder where that leads. I guess we are very dependent on pretty much every single person in these special reviewers being some sort of super honest person
 that has no incentives and also no strong opinions on these things and and generally has gone into this ML ethics just out of interest and not to actually make an impact.
 I'm sure that will work out just fine.
 Now, the official NURiBS website actually links to a blog post of Brent Hecht, suggestions for writing the NURiBS 2020 broader impact statements.
 So we can reasonably assume that this is at least in agreement with the organizers of the conference.
 Brent here says understanding the societal impacts of your work is going to be hard.
 It is going to take lots of effort to write NURiBS broader impact statements. Tons of work has already been done for you.
 Check out the literature from communities that have studied societal impacts of AI for a long while.
 Even better, bring a social scientist onto your research team. Remember, though, they don't work for free.
 Hire them into your company, give them subawards, recruit them as PhD students through interdisciplinary programs.
 So are you saying the more problems these people find, the more of them will get hired?
 And again, look at these statements in general. It seems to really be about how much work are you able to put into this?
 So here's your average PhD student.
 Now, they pretty much already have to write their papers by themselves or in very, very small teams because they need first authorship.
 And they have to do all their experiments and they don't have enough resources.
 And now they're also asked to spend considerable amount of time not only writing this very hard statement,
 but also reading up on all the literature that there is to read up on.
 Or alternatively, if you don't want to do that, well, just hire someone, of course, because budgets and salaries in universities and for PhD students is notoriously loose.
 We can just hire someone that does that.
 I don't really think it's possible for single PhD students or research labs at universities to just hire someone or put someone full time on this additional required work that is put onto them.
 I wonder who will be actually able to put additional people on this such that they surely end up with beautiful, well-researched, broader impact statements to justify even the most ethically concerning research.
 I'm wondering, it just gets on the tip of my tongue, but I was going to have to leave that for another time.
 For people who do more theoretical work, it is going to be more difficult.
 Wait a minute. I thought the official communication was you're very free to leave it away if you don't think this applies to you.
 But here it basically saying it's going to be more work for you.
 Find something that is both rigorous and practical for your research.
 And the argument is to get funding for any theoretical work, someone had to make an argument about positive societal impacts at some point.
 Not true. Some universities simply get money and academic freedom.
 If that argument is possible, it is probably also possible to make a rigorous statement about some negative societal impacts.
 You might be tempted to write boilerplate low information statements.
 Don't do this. It will undermine the rigor in the rest of your paper.
 The public will roll its eyes and reviewers may and often should call you out.
 Now, wait a minute. I thought the reviewers are only supposed to judge the adequacy and absolutely have no influence on their judgment of the paper.
 But the underpinning of this text here is basically that if you as a reviewer feel that this hasn't been done adequately, you sort of should let this swap over into your assessment of the rigor and adequacy of the technical contributions in the paper.
 Because they're kind of the same, right?
 And they also say that this might spark a conversation is specifically the author response period will be a decent opportunity to have a bit of a dialogue between author and reviewer on the impact statement.
 So this basically means that I as a reviewer now have to write in my review something about the broader impact statement and not only judge its adequacy in a special field for it.
 And then the author is forced to spend a bunch of their very, very, very valuable author response on rebuttling the reviewers assessment of their broader impact statement.
 Our proposal's view is that it's not your job as a reviewer to judge submissions for their impact.
 Rather, you should evaluate the rigor with which they disclose their impacts.
 So again, it's about putting work into it.
 If you go to the full proposal behind this blog post, you'll find the following snippets.
 So there is a list of expected outcomes of introducing such mandatory broader impact statements.
 Now, have a look at this expected outcomes.
 We expect that action on the above recommendations will lead to a number of desirable outcomes.
 And they're all positive outcomes.
 Now, haven't we been discussing for the last minutes that it's always important to assess the positive and the negative outcomes of your actions and of your releases?
 How ironic that none of the organizers of the conference, nor any of these people communicating, were forced to release a broader impact statement discussing the negative consequences that their actions would have on the community and the greater society.
 They go on to give a list of examples of how you could do such positive and negative aspects of technology.
 One of them is social media.
 And I would agree that there are ethical considerations if you invent social media.
 We all know that social media can be some sort of a dopamine feedback loop addiction and have negative consequences in society that are not readily visible.
 But it goes on. Crowdwork, a researcher who invents a new crowdwork framework, likely motivates her work by highlighting the problem the framework solves.
 But they go on to say that crowdwork also has negative externalities, such as incentivizing very low pay.
 And the researcher should find ways to engineer her crowdwork framework such that these externalities are structurally mitigated and or she might advocate for minimum wage laws to be adapted.
 So this researcher working out a problem in crowdworking now has to basically solve millennia old problems in structural economics that have thousands of moving parts and no clear consensus on how to solve.
 But the best example, and this is the example they actually tell you to look at if your work is more theoretical or you don't really think it has an impact, is the following storage and computation.
 Recent advances in storage systems and graphical processing unit processing afford the easy storage of massive amounts of data and the real time computation on these data.
 This has incentivized corporations to collect every possible data point about their users, save this data indefinitely and strive to monetize this data in new ways.
 While allowing for impressive new capabilities, this trend also presents tremendous risks to privacy.
 Researchers working in storage and GPU processing should consider these and other foreseeable potential risks in their papers.
 They should also enumerate technological and policy means by which these risks might be mitigated, e.g. technologies to automate general data protection regulation require capabilities and improvements to GDPR like policies.
 That is absolutely mad. So here you are making a GPU chip more powerful and you're asked to think ahead about the fact that this can be used to mine data.
 And not only that, now you're also required to propose improvements to GDPR like policies.
 The GDPR only an 88 page, very fine print legal document that in addition to all the literature about AI governance, our poor PhD student is now also required to read, understand and be able to improve.
 How long does this chain of causality go? How do you have to think ahead? This gets ridiculous.
 It's 200,000 BC and Nuno in his cave just invented fire.
 Well, fire can be used to cook food, can be used to have less disease, can be used to settle down, expand civilization, build educational facilities, build up a culture, a scientific method, enable massive progress, industrialization,
 general improvement in health, wealth, education and happiness of society, which ultimately leads to some people building GPUs and saving your data and analyzing all your things to serve you ads of better kitty stickers.
 How could Nuno in his cave do this to us? Where is his broader impact statement about the invention of fire for the future data collection algorithms on GPUs?
 Look, I'm not saying that you should not consider the downstreams effect of your inventions. Of course you should.
 But at some point it gets ridiculous for most of the work handed into a conference like NURIPS.
 Either the downstreams effect are so far away that is almost impossible to foresee or as any technology you can use it for good and for bad.
 And it is going to be with the application of this technology and not its invention where the good and the bad come in.
 And what most people are going to do is simply come up with things that mean absolutely nothing and generally make not a lot of difference.
 While it gives a big advantage to big institutions that can spend a lot of time and effort on crafting very rigorous adequate statements.
 Another release called A Guide to Writing the NURIPS Impact Statement that is not linked by NURIPS, but as they say was in communication with some of the organizers of the conference.
 So it's reasonable to assume they also largely agree with these positions here.
 Says you should discuss, read and reflect time permitting impact assessment will benefit from broad intellectual reflection, discuss potential impacts, follow public discussion, read case studies and read the scholarly literature.
 On tech governance, of course, time permitting.
 But then again, if it's not rigorous enough, a reviewer might be getting the idea that the rest of your paper isn't rigorous enough.
 So maybe time must permit for this one.
 And they again say, think about impacts even for theoretical work.
 So the official communication always says that if you don't feel this applies to you, you're very free to write.
 This doesn't apply to me.
 But the in official communication says if this doesn't apply, you're doing something wrong.
 And by the way, we're evaluate the rest of your paper based on the amount of work you put into that statement.
 Ultimately, these statements are just going to boil down to you can do good and bad things with any technology as is visible on this example they give here.
 Pluribus, a superhuman AI for multiplayer poker, they say they intentionally choose to broaden the focus of their broader impact assessment, depending who can use this scientific advance such as criminals or well motivated citizens.
 This technology may be socially harmful or beneficial.
 If access to this capability is mostly available to the wealthy, it could plausibly promote concentration of wealth.
 And further on the other side, increased skill could increase total welfare.
 Gee, if that doesn't apply to every single technology ever, I don't know.
 Again, my general assessment of this is not that it is absolutely wrong to do this or very useless.
 It is just shifting the balance a bit more on to large institutions who can actually afford to spend a lot of time and work into crafting beautiful statements.
 And in general, I don't think it's that big of a deal, but I also don't think it's going to help very much to just force everyone to do this.
 I guess we'll see how it turns out.
 In this VentureBeat article, they link someone named Joe Redmond saying, I stopped doing CV research because I saw the impact my work was having.
 I love the work, but the military applications and privacy concerns eventually became impossible to ignore, which I respect a lot.
 But I would ask, did Joe Redmond realize this after being forced to write a broader impact statement or at some other point?
 That was my two cents.
 If you like videos like this and paper analysis and other things, then subscribe, like wherever these buttons are, share it with your friends and see you next time.