If Ethics is not None
Jarmul, Katharine
If Ethics is not None The history of computing, as it's often covered in textbooks or talks, remains primarily focused on a series of hardware advancements, architectures, operating systems and software. In this talk, we will instead explore the history of ethics in computing, touching on the early days of computers in warfare and science, leading up to ethical issues today such as Artificial Intelligence and privacy regulation
 All right, so now I want to welcome you, our next keynote speaker, it's Catherine Jermul. First time I saw Catherine was actually online on videos recorded at PyCon US where she was talking about web scraping and data mangling and I really loved her talks. Finally, we got to know each other actually in 2015 and as you know, like she's an author, she wrote a book on data mangling on a Riley, she does video courses, she's one of the organizers of PyData Berlin, she's a very active member of our community and now she's going to talk about ethics and data... Oh no, sorry, you're basically also like you're European basically, she was born in LA but you're in Berlin since three years? Yeah, about three and a half. So and today she's going to talk about ethics and data, so this is not a technical keynote, it's an ethics keynote, so I want to give you a very warm welcome to Catherine. Thank you for being here, I know there was quite a fun social event last night, so I appreciate you getting up early and joining us and I want to thank all of the organizing committee for organizing such a great conference and also inviting me to be here. Today I'm going to talk about a topic that is quite controversial in some circles. Now ethics might be something that you learned about in school, it might be something that you even debate on the world stage or when you talk about politics with your friends, but it's not often that in the work that we do every day we get into ethical conversations. But I wonder if it should be, I myself work as a data scientist and I myself face these types of ethical conversations often when I'm building models or when I'm thinking about how we use data and because of this I think that perhaps us even as a larger computing field even if you don't work with data, perhaps this is something that we should be talking about. So what we'll do today is I'm going to take you through some of the history of computing with an ethical focus. So we're going to take a look at important moments in computing history and some of the ethical reasoning behind them. To begin with we will start with an IBM advertisement from 1960. This is where Americans need to mind their game. Around the clock radar electronic eyes watch those guys and report what they see to say the vent system of the United States Air Force. Here it is saying center on. Oops sorry. I don't know how to make this. Let's go away. All right. We'll actually move forward because in the interest of time and in the interest of all of your time what this is an advertisement for and what it goes on to explain is the SAGE system and this system is a satellite air system and what it is supposedly doing is tracking all of the aircraft over the United States and its goal is to have these people, these intelligence workers to be able to click on a screen at these unknown aircrafts and then fire missiles at them and destroy them midair. Therefore protecting the United States from quote unquote Soviet attack. This was a system that was designed by a lot of intelligent computer scientists. A lot of people that were building IBM's best computers at the time in the late 50s and early 60s. And when I saw this advertisement I mean it's quite over the top. I will have to post it later so you can look at it yourself. But I wondered what the computer scientists who were working on this machine thought. I wonder if perhaps they were at all concerned ethically about their creation. And in testing actually this SAGE system was wholly ineffective. In one test called Operation Sky Shield it would have only neutralized 25% of targets leaving 75% of missiles incoming and to whatever devastation they were able to make. And so I wonder if when they saw this advertisement and it's made to seem like ah you're completely safe and secure because we have a computer and the computer will take care of it all. I wonder what the computer scientists actually thought the computer engineers who were building this system. And I wonder if they were concerned with how it was marketed and if they were concerned about failure. So this is a constant problem in computing. Computing has been touched by the military, has been touched by state intelligence systems in a lot of ways throughout history. The first computer scientist and mathematician we'll look at is Norbert Wiener. And Norbert Wiener was a famous statistician in his time and he worked with neurologists and other mathematicians to help discover some of the ways that our brain sends electricity. So this is when we were first discovering how electrical signals travel through the brain and when we were first discovering neural networks. And he also did some statistics for the military. In fact his work went on to contribute to the minimal mean squared estimator which essentially allows us to estimate for he used it for flight paths for missile defense. So he himself worked with the military, he himself probably had his own ethical qualms. But his seminal work is a book called Cybernetics and he was actually one of the first to coin this term. This is this idea of using computers as an aid to help us make decisions. And in Cybernetics he has a quote that I think is important to share today. I'll read a slightly longer version than we have here. I have said that this new development, computers, has unbounded possibilities for good and for evil. It gives the human race a new and most effective collection of mechanical slaves to perform its labor. Such mechanical labor has most of the economic properties of slave labor. Although unlike slave labor it does not involve the direct and moralizing effects of human cruelty. However, any labor that accepts the conditions of competition with slave labor accepts the conditions of slave labor and is essentially slave labor. The key word of this statement is competition. It may very well be a good thing for humanity to have the machine remove from it the need of menial and disagreeable tasks. Or it may not. I do not know. When I read this quote and I realized it was written in 1948 I was a bit surprised that I don't feel like we've moved much further along in the conversation. I feel like this quote could be shared in an article today, in a blog post today, in any type of news today as we debate automation and jobs. The people in our field we work to automate tasks away. Sometimes we work on systems that will entirely replace an industry. And therefore are we then responsible for the degrading of lives that happens when that occurs? Perhaps it's better that nobody has to be a factory worker or that nobody has to drive a taxi if they don't want to or be a truck driver and so forth. But what does it mean? Is it the tyranny of the few? Is it perhaps those of us that can automate away things and the societies that can afford to automate away things versus those that perhaps can't? And if those other societies or if other nations can't keep up does this mean that they're essentially competing against these robots against this as he so-called slave labor? The next mathematician that we'll look at and great computer scientist is Joseph Weisenbaum. Now Weisenbaum was essentially a contemporary. He was a German Jew and he was born in Berlin. His family escaped Nazi persecution by emigrating to the United States and he grew up and was educated primarily there. He became a professor at MIT in the 1960s and you might know of his work because he built Elizabot. So for those of you familiar with natural language processing or perhaps you've just heard about it, the Elizabot was probably the first chatbot and it used natural language patterns to essentially mimic human speech and act as a therapist. Now Weisenbaum was also famous in his own right for some other inventions that he did. In fact, this is a photo of him at DeSite in 1965 and he's demonstrating remote login to his MIT machine. Weisenbaum came to become quite active politically within his time and he actively worked to challenge those around him on these ethical and political concerns that he had. I think he felt it especially important given his family's history to talk about ethics in computing and what he saw around him. And he began to question whether AI and computer science were a force of good in the world. This is a quote from an interview in an MIT publication in 1985 where I feel like he was particularly candid about his feelings about computing. I'll again read a longer quote. I think the computer has from the beginning been a fundamentally conservative force. It has made possible the saving of institutions pretty much as they were, which otherwise might have had to be changed. For example, banking. Superficially it looks as if banking has been revolutionized by the computer, but only very superficially. Consider that say 20, 25 years ago, the banks were faced with the fact that the population was growing at a very rapid rate. Many more checks would be written than before. My response was to bring in the computer. By the way, I helped design the first computer banking system in the United States for the Bank of America 25 years ago. Now if it had not been for the computer, if the computer had not been invented, what would the banks have had to do? They might have had to decentralize or they might have to regionalize in some way. In other words, it might have been necessary to introduce a social invention, not just a technical invention. And this quote for me brought me pause as somebody that aims to make unmanageable tasks manageable. I aim to make data clean and accessible to use large and disparate data sets to make inferences or determine some sort of meaning or signal. Am I helping consolidate power? By doing the job with maybe 30 lines and a scikit-learn import, by doing the job that perhaps would be very untenable, take quite a long time, or maybe even impossible, am I therefore helping consolidate this power? Am I using technology to thwart social progress? I don't know the answer to this, but these are the types of questions I'd love to hear your feedback on. Moving on throughout history, we have Oliohan Dal and Christian Nygaard. They're essentially the fathers of object-oriented programming. So we have them to think. Simula 1 and Simula 67, the languages that they developed, were computer simulation languages. They were used to simulate physics in a military laboratory. And Christian Nygaard here on the right was a staunch leftist and activist throughout his entire life. In fact, he eventually identified as a socialist. In his belief system, he of course believed in supporting workers' rights, and so when he was approached by the Norwegian Iron and Metal Workers Union to help them build a system so that their jobs would not be automated away and instead they would learn computing skills, he jumped at the opportunity. And he was able to work with them and build what many people think is the first example of participatory design, one which the active participants or users or workers are able to help make the design process and able to help determine what the system looks like. And he essentially helped them build an operations research type of scheduler and backlog organizer for the union. He was also quite famous for walking away from military work. In 1960, he quit his job at the defense research in Norway and took several of his teammates with him. He often joked that because of that, he had the most funding request rejections out of any person in Norway. He gave a speech at the Iris Conference, which is a group of Scandinavian researchers that meet. And I'll read a slightly longer quote from it. You need a self-defense against yourself and the temptations to choose a comfortable but wrong way out in critical situations. But compromises may be necessary. The greatest danger then is not the acceptance of a dubious compromise, but in not being cynical and honest about it. Your mental processes will try to justify your actions to yourself, making the compromise the desired solution. And you will change yourself if you're not honest and astute. Bo Dalbom asked me to talk about the iron and metal project. Why? Many people don't know about it properly, he said. And some have forgotten those aspects that ought to disturb them as their environment pushes them slowly to the right. Perhaps I should ask some questions to those in the audience who believe that they have been influenced by the project and its successors. Has anyone resented the content of your work recently? If not, what is your excuse? Have you had any real conflict in your research activities lately? Or does such conflicts only belong to your now romanticized glorious radical past? Will your recent research to any extent increase the power to influence their own fate for people with whom you feel solidarity? He went on to quip that he's joking a bit, but I think the message remains clear. If we're building systems, tools, algorithms, and so forth, if we're building these and they actively work against our politics, against our ethics, against our morality, however it is you choose to make valuable decisions in your life. If we do that, why are we doing that? And if we choose instead to build tools and systems that we believe support our politics, that we believe support our ethics, and we distribute them, can we be then a form of justice? Can we therefore spread instead of spreading, let's say, something we don't believe in, can we therefore become a way to spread something that we do believe in? Or as he suggests, are we just constantly perhaps making compromises, and do we need to be very honest with ourselves about the compromises that we are making? Andrei Urshok is a prominent figure in Soviet era computer science and cybernetics. He worked on numerous inventions along the same time, working in parallel and often collaborating with several computer scientists in the United States and the UK. He's actually famous for likely writing the first optimizing compiler for a language that's more complex than Fortran. He wrote this for the alpha language that he worked on. And he was a really big proponent of education. In fact, he's probably the first computer scientist to talk about computer literacy. He built several schools during his time, he appropriated funding for them, and this, for example, is a photo at the USSR Academy of Sciences during a summer school for young programmers. His belief in teaching and in learning, and that being an important part of computer science, is really espoused in his speech titled Aesthetics and the Human Factor in Programming, which he gave at a 1972 computer conference in the United States. I'll read the full quote of which only a part is shown here. In past ages, the ability to read and write was considered a rare, God-given talent gift. The destiny of a limited group of the specially chosen. In the present age of general literacy, we perceive reading to be a universally attainable accomplishment, but we are tempted to single out a new elite group who become arbiters between the lay generality of mankind and the arcane informational model of the world hidden in the machine. Is it not, however, the highest aesthetic idea of our profession to make the art of programming public property, and thereby to submerge our elite exclusiveness within a mature mankind? Indeed, I feel like Urshav approaches these ethical problems slightly differently than some of his Western contemporaries. His idea is not, do we hold some power that we should then further hone for good or for evil. His point is, should we even hold this power at all? Should everyone learn to code? Should everyone learn how a computer works literally on the inside, not just how to turn it on and off? And if not, does our own understanding of computers, does our own ability to code, does, for example, the data scientists in the room, our understanding of statistics and machine learning models, does this give us some special power or privilege? And if it does give us a power or privilege, are we an elite? And what is being part of an elite mean? Does it mean we have a responsibility? Does it mean that we have to do things differently? Moving on to the networking era, which is always fun, right? We all love the Internet. Here you can see the employees at BBN, which is Bolt, Baranik and Newman, a Cambridge research firm. And BBN won a contract with the Advanced Research Projects Agency, otherwise known as ARPA. And their project was to build the first computer network in the United States. They're seen here in 1969 along with their interface messaging processor machine, which is essentially the first router. And in case you don't know, this is the same group that started to create the protocols and the standards that we use in today's Internet. There's a bunch of amazing engineers and architects that I could talk about in this photo, but I'm going to focus on two of them. Here in blue, I have several, and in yellow, I have Bob. Why just talk about the two of them? Well, first and foremost, because they went on to have longer careers and do quite a lot of work after ARPA. But secondly, because they were very outspoken in their political and ethical beliefs, and they went in quite different directions. So I think it's perhaps a good case study to explore the life of a computer scientist. We'll focus on Bobcon first. Bobcon worked on networks and essentially networking protocols. He along with Vincent Surf was able to create the TCP and IP protocols and standards, which are mainly unchanged today, which is pretty amazing, feet. He's essentially a networking genius. He went on to build very large networks for the US military at DARPA, and then after that he went to IPTO, which is the Information Processing Techniques Office. Then he heard about this new project brewing in Japan called the Fifth Generation Computer Project. And this idea was how can we create true AI thinking machines? How can we create these decision support systems that we might be able to use? At that time, he then pitched the idea of the strategic computing initiative, of which this is one of the plans. So for the strategic computing initiative, you have, as you can see at the bottom, all of the infrastructure. These are the networks that Bobcon loved. This is something that he loves to work on. He was able to get quite a lot of funding to just build networks and come up with new networking concepts. On top of that, they would then try to build some chips, some different hardware designs, and eventually working to natural language processing and speech and vision recognition and so forth. On top of those technologies, they would build autonomous systems, pilots associates, and battle management. And this was all to, and I quote, develop a broad base of machine intelligence technology to increase our national security and economic strength. So what does this mean? I mean, when I first saw this diagram, I said, well, perhaps Bobcon didn't even know about how his networks would be used. Maybe ethically for him, he just really wanted to work on networks. I can understand and appreciate that. And then I found a quote from around this era of Bobcon. And it states, the nation that dominates this information processing field will possess the keys to world leadership in the 21st century. As I began to read more of his writing during the time and writings of his peers at SCI, for example, Lynn Conway, who is a prominent computer scientist in her own right, it was clear that they were very well aware of how their programming would be used. In fact, it's rumored that the chart that we saw in the last slide, that Bobcon himself designed that pyramid. So it's clear that he knew he was building weapons. What I wonder is, what does it mean when we have to take military funding or let's say corporate funding of corporations, maybe we don't agree with, to do our work? Does this make us ethically culpable in what the greater process is? And if it does make us ethically culpable, if something goes wrong, let's say civilians are bombed and so forth, does that make you as the computer scientist or as the data scientist or the machine learning expert, does it also mean that you are culpable? And if you aren't culpable, then who is? Going back to Bobcon's peer, Severo Ornstein, he was a co-worker again and a peer at ARPANET. And here we see him with Laura Golt, who was a computer scientist and activist as well. Severo's work at ARPANET was as lead hardware expert. So he built most of the hardware for the first router and also did some of the software programming. He later went on to join Xerox PARC and he worked on the Dorado, which was the fastest processor at the time in operating system. So around the same time that Bobcon was petitioning the government for millions of dollars to build weapons, Severo Ornstein and Laura Golt were starting another initiative. And this was for a group called Computer Professionals for Social Responsibility. This is one of the patches that the Computer Professionals for Social Responsibility gave out at a joint AI conference, most likely the one in Los Angeles in 1985. And some older folks in the room or perhaps some that knew a little bit of Americana history. This is a play on a famous Cold War era advertisement, if you will. It was a public service announcement that came on and it would say, it's 11 p.m. Do you know where your children are? And its goal was to enforce curfew and to make sure that all the young people were inside at night in case of danger. Now this play on it is it's 11 p.m. Do you know what your expert system just inferred? And we see the telltale mushroom cloud of an atomic bomb. And these are the same expert systems that Bobcon himself was trying to build. I mean, they never really came too much, right? At that time it was very quickly before what we all know now as AI winter. And these expert systems were never much more than large statistical models. But the fear at the time was that they would be treated as experts. It was that if the system itself said, okay, we're under attack, it's time to launch the nukes, that that is something that would actually happen and that a human would go about doing it based on this expert system. That was a massive concern for CPSR, the computer professionals for social responsibility. Now CPSR was around actually until 2013. And they gave out fairly regular awards actually named Nelvarth Vina and highlighting people in the field who are working on social responsibility issues as computer professionals. It's a pretty amazing list. We've actually covered some of the people on the list, but it goes on to talk about whistleblowers and other people who really worked actively to act as an ethical agent. But I think what we can really take away from CPSR or what I took away from CPSR is that it was starting this type of activist conversation amongst computer professionals. And it was asking, what can we do as people who are concerned about the world, that as people who are concerned about social responsibility, how can we help? How can we be a part of change or a part of something that we believe brings justice to the world? And I think that it's quite sad when you read about the history of CPSR that around the time of the personal computing revolution in the mid to late 80s and early 90s, they saw their numbers dwindle. And I think one of the problems of this is computing went from being a social experience in the computer lab, working with their peers, chatting probably a lot about these types of issues and theories. It became a personal and a lone experience. This was also the time that is often tracked to the gender disparity within our field. That's often tracked that the advertisements were all of a sudden, be a gamer, it's really cool for boys to have computers. And so I feel like this was perhaps not intentionally, but unfortunately, one of the fatalities of that era was this conversation about computer professionals for social responsibility. But CPSR was not the only ones fighting for more professionalism in the field or fighting for us to be having these conversations. You might know of Karen Spock Jones. She's quite famous. She's a statistical, she's a statistician as well as a natural language processing expert. And she worked at the Cambridge Language Research Unit on a variety of NLP research. Probably her first forays, I guess, are into looking at inferring meaning from the Sari definitions in text. You might have also heard of TFIDF, term frequency, inverse document frequency. It's a popular way to extract important words or phrases from a text when you have a larger corpus. She wrote the paper that defined the IDF portion of the equation. So she's pretty important when it comes to natural language processing. But that's not her only contribution. Spock Jones was an outspoken critic of the gender disparities within our field. She also was quite a critic of not having some sort of professional code. She thought that the growing importance of computers in people's lives meant that we had more and more influence and she was concerned that we didn't have any type of licensing board review, that we didn't necessarily have any standards for our own code, our own actions. I'll again read a slightly longer quote. I certainly think that professionalism is very important. To be a proper professional, you need to think about the context and motivation and justifications of what you're doing. You don't need a fundamental philosophical discussion every time you put your finger on the keyboard. But as computing is spreading so far into people's lives, you need to think about these things. I've always felt that once you see how important computing is for life, you can't just leave it as a blank box and assume that somebody reasonably competent and relatively benign will do something right with it. For a quote, again, evokes this idea, who is responsible? If I build a program or write a script or build a model that does something unethical or that can be used unethically, am I responsible for when somebody does that? If I'm not responsible, who is? As especially machine learning and numerous other fields that we work in as they touch more and more lives, as they're used even by legal experts or by the police or by state intelligence or by doctors, are we not also beholden to these same licenses, these same ethics that these fields have? Now, I'm not necessarily saying we need to have a licensing or we need to have a particular code, but if we have no standards, if we have no way to review one another's work and to think about whether we're falling in ethical principles or whether we're helping or hurting lives, then we don't even have a starting place for these conversations on professionalism. And if we're not willing to take on the burden of thinking about these things, then does the burden actually go away or are we just ignoring it? Finally, we've reached some of the outspoken leaders of today's AI in today's era. This is Joanna Bryson. She is a professor of computer science who speaks on AI and natural intelligence at the University of Bath. And she's been working on the field of AI ethics since the 90s. She is a prolific writer and she's published all the time. She has a paper over this past year on the epics of looking at word vectors and different gender disparity within them. But she also has some older works. For example, her paper, Just Another Artifact, Ethics and the Empirical Experience of AI, articulates this problem of humans over identifying with artificial intelligence. She essentially says that instead of treating it like a book or a reference, we treat AI as a human actor, as another intelligent human being with reason and common sense and so forth. And she states that this is quite dangerous. If we start treating AI as more expert than ourselves, if we start treating it as the sole singular expert in the room, then what happens when it makes a mistake? So she's been quite outspoken on these issues. On one of her longer posts, she articulates some of her well thought out opinions on how we use data. I'll read again from a longer quote. As we in the computational social sciences learn more and more, our models of human behavior get better and better. As our models improve, we need less and less data about any particular individual to predict what they're going to do. So just practicing good data hygiene is not enough, even if that was a skill that we could teach everyone. My professional opinion is that there's no going back on this. But that isn't to say that society is doomed. Think of it this way. We all know that the police, the military, even most of our neighbors could get into our house if they wanted to, but we don't expect them to do that. And generally speaking, if anyone does get into our house, we are able to prosecute them legally and claim any damages back from insurance. I think our personal data should be like our houses. First of all, we shouldn't be seen as selling our own data, just leasing it for a particular purpose. This is the model software companies are to use for their products. We should just apply the same legal reasoning to we humans. Then if we have any reason to suspect our data has been used in a way we didn't approve, we should be able to prosecute. That is, the applications of our data should be subject to regulations that protect ordinary citizens from the intrusions of governments, corporations, and even friends. What Joanna Bryson is asking of us here is to actually be better than the current laws. Now we could get into some debates about the new EU privacy regulations and the fact that yes indeed it has some ways that I can control my own personal data and I can request information about how my data is used. But those are clearly not global and they have yet to be determined in the courts how they'll be enforced. What she's asking us to do is to treat people's data like their property. Would you buy and sell another person's property without them knowing it? And if not, why do we do this with customer data? Would you collect pieces of another person's property without properly informing them in plain English so that they can understand what they're agreeing to? And if not, why do we do it in our terms and conditions? And a particular concern of mine regarding security, would you take your friend's laptop or computer or books or whatever it is that they have and would you leave it on a public table outside for everybody to walk by and perhaps look at? And if not, then why are you leaving your databases with protected information or your models with protected information on the public internet? Debating ethics at work is nothing new. My great-grandmother was a pretty amazing woman. Here's her. Her name was Adele Matilda Rich. Her on the left is her growing up in New York City and she used to tell me really fun stories in the 20s and 30s of growing up in New York City. And her on the right is her with a very young version of me. Now, Adele, as I used to call her, was amazing. She has a lot of good memories that I have in my mind. But the reason why I bring her up is the fact that she worked as a secretary on the Manhattan project. The Manhattan project was the project in case you don't know the code words for the nuclear bomb in the United States. And she worked as a secretary. She used to tell me fun stories that she did something important for the war and that she had to burn her notebooks and so forth. And I just thought it was a fun story until I got older. She died when I was a child, so I was never able to have these types of conversations with her. But I did talk with my mother, who I know had several conversations. She said that Adele at first didn't know exactly what she was working on, but slowly put the pieces together over time and then of course when the bombs were dropped, she was quite aware. And I asked my mom, did she ever regret it? Did she ever stay up at night wondering if what she did was right or wrong? And my mom said that she thought that Adele felt like many of her generation that she was helping the good guys and that any deaths that were created by the atomic bombs dropped were just small casualties in a larger war. I wonder if I could go back and ask her if we could have a longer conversation about this. But it also makes me wonder if my own privilege now, something that she worked so hard to do. She worked to help make sure our family could survive through the war and the Great Depression. And she made it so that my mom could be the first one to go to college. And my mom going to college meant that I was able to get a computer and that I was able to go pursue computer science. So maybe it is wrong of me to question something like that. Perhaps she was not in a position to make a decision between ethics and feeding her family. And perhaps it is rather ignorant of me to even ask that question. But I also wonder if it is not a right for all of us to have rewarding work that's both challenging mentally, financially rewarding, and also ethically something that we can support. So I challenge us as a community. As data scientists, web developers, if you work on MicroPython, CorePython, Jython, whatever it is that you do, Internet of Things, I challenge us to start having these possibly more difficult conversations. I challenge us to start thinking about who is responsible and how we can hold one another accountable. Now, you might not work on anything that touches ethics, but perhaps you do. Or perhaps a coworker does, or a friend, or a colleague, or somebody you meet at this conference, perhaps a future you, or a past you. I know in my career I've been definitely asked to do unethical things. And I've been asked to lie, to make charts look different, to make data look different for the greater good, right, for extra funding, for whatever it is that the goal of the company or the product is. So I've had to have some of my own ethical decisions. I think that hopefully what we've heard today and what we've shared today can start to be a ground for a communal conversation. I know that in a community like this one, we have values. For example, I feel like as a Python community, we value diversity. I feel that we value free software and open source, and I'm very proud of that. I also feel welcome and supported today, just the same as my first Python conference in 2010. So we have these shared beliefs and these shared values already. But perhaps we don't have a larger understanding of how they relate to ethics in computing as a whole, or how they could inform a communal ethic, or communal principles, codes of code, and conduct, ideas that we share with one another, and conversations that we have openly. Now it may not be that we come to some communal agreement, but just the idea of sharing our stories openly, sharing stories like these from around the world openly that tell us a little bit about the history, the ideas, and these deep questions and dilemmas that we have. Perhaps that is a way to start being a better force in the world. We data scientists, programmers, Pythonistas, computer and software engineers, we do have this cultural and ethical history. I hope that it's shown this is only just a few of what I was able to find in my research. I want us to share these stories, and I want to hear your stories and have you share your stories as well of these ethical dilemmas that we face and the decisions that we make around them. And I truly, truly believe that in doing so, we are more. That when we have these conversations together, we are actually contributing to a greater good. Perhaps by having all of these conversations and working together, we can make sure that AI, machine learning, natural language processing, and Python have a bright future in justice. And that that's something that we can be really proud of. And I know that I want to be a part of that, and I hope that you do too. Thank you so much for being here, and I hope to continue this conversation throughout the conference. Thank you very much, Capric. I think it was an amazing keynote. Now everybody is really quiet because there's a lot of thinking going on in the room, but we have time for questions. So who has questions or comments? And if you're able, it would be nice if you could talk about that. Thank you very much. I have a question about, we'll just say, banks. There was no computer that they could call different. But thanks to the computers, we have automated ways right now, like PayPal, and the cryptocurrencies, which basically allows us to operate without banks. So what do you think about this? Yeah, so I definitely think I shared some folks today throughout history, and there was a few that touched a little bit on this, but I feel like this is definitely something more of our generation, our era, the idea of using kind of some of this technology to subvert the norm, right, to actually actively work against corporate influence on our life or government influence on our lives. So I think that there's a lot of ways that technology can be used to kind of actively work against some of these ideas. I completely agree, and I think that that's a really important conversation that unfortunately didn't quite fit in in this talk, but I think that's really important to share and think about. And this is kind of this idea of us creating technologies that actively support what we believe in, and then as we spread them in the world, that that hopefully makes more things possible. Like, I look at teams like the Signal team and what they're working on, and I think it's really great. There's some amazing work out there, and perhaps you work on those things as well, and these are really interesting conversations to me and things that we should also be highlighting, right? Thank you. Hi, thank you for the amazing talk. I would like to ask you a question about auto-automatization, especially if you see difference between what will happen probably with this machine learning revolution in some sense, and something like the industrial revolution with what happened in China, because the advantage that I see, or those other things that happened in the past, is that in fact they lifted a lot of people from poverty, even though it was not a great process in some sense, that was the main advantage. One should always say when you talk about, you know, auto-automatization machine, it has also human side, which was helping people. So do you think there is some differences between what will happen and what has already happened and why this should raise some different question regarding ethics? So, Zai, the question exactly is about how the computer revolution is different than revolutions before it, or, sorry. And the question is how do you think ethically this is different in some sense, one should be what we should think about? Is this just automatization on a larger scale, or is something, in a different way? Yeah, I wonder about that, because, yeah, clearly the industrial revolution was, and other previous revolutions, for example, the printing press and so forth created their own ethics and their own issues. What I wonder about with computers is, like as we've seen with Moore's Law, and then also with the new post-AI winter, we're seeing quite a lot of progress in a short amount of time. I don't know if that's the same amount of progress. I obviously wasn't alive then, and I'm no expert on previous revolutions. I'm not even an expert on this revolution. So I don't exactly know, but I wonder if the pace and perhaps the expanse is slightly more impactful now, that maybe because of the internet and because of the advances that we've made in computing, perhaps we have a more quick impact on people's lives. I don't know if this is true, but sometimes I think that perhaps this might be the case, but I would be curious to hear more of your thoughts on that topic as well. Great talk, really great. I was thinking about the ethical side of what we do on a daily basis is really important, but my concern is more, for example, the people who started investigating the outcomes and their subparticles many, many years ago, they had no idea that this will lead to the Manhattan Project. So do you have any advice on how we can try to predict the future implications of our work and how they can, the ethical result 50 or 100 years from now? This is something where if I had an answer for you, I would be telling it to the world from the rooftops. And I don't know, I think perhaps we need to, I echo that concern. I don't know what things that I work on today and things that people much brighter than me work on today. I have no idea what large impact these will have on the world a decade from now, even five or 10 decades from now. But I do think we should be thinking about it, and I think that I was heartened to when I was reading and doing research for this, that some of these people were trying to project and think about what this could do in the future. Now a lot of the times they were pretty far off. But I think that it doesn't hurt to start the conversation now and to work together to perhaps see what this might mean in 10, 50 years. And for example, how we can protect against things that we're fearful of and also support ideas that we think would be really great things that we can do with technology. Thank you for an awesome keynote. I think at least from my point of view, it's pretty obvious that automation of work is a trend that will continue and that will probably worsen any social disparities that we have now. Do you have any ideas where to start a conversation about how technologists can help in this situation instead of worsen the state that we are already in? Yeah, so I think that as I was thinking about this and when I found the computer professionals for social responsibility, now there are some groups that are working on these types of things. There's like fairness, accountability, and transparency, machine learning group and conference. There are some other ones that are trying to start these conversations about how we can be socially conscious actors when we do automation. I think that this would be a great thing to have a group around or a conference around or ideas around. I am no expert. I am just a person like you who is concerned. And I think that the more that we are able to share our ideas, I think that honestly, collectively, we are much more smart than any of us individually. And perhaps by having these types of debates openly, we can figure out maybe thoughtful ways that we can automate so that perhaps the impact is lessened. I also agree automation is definitely not going to go anywhere and nobody is going to stop automating because of a keynote or because somebody has concerns. So I think that, yeah, having these conversations and collectively coming up with some good solutions or some good ideas is most likely the most powerful thing that we can do. We have three more questions. And if you promise to keep them short, I'm going to take them. But I think probably since it's such an interesting subject, would you be willing to do an open space for that? Yeah, sure. Sure. So basically, if you don't know open spaces yet, we have a large room there. You can just join there and make up your own EuroPython. And there's a, yeah, that's what it's for. So there's a whiteboard downstairs. You can use it. So we're going to put an open space for this on. I'm going to tweet all also about it because I think many are interested and there's like three more short questions. Yeah. Yeah, cryptocurrencies have been just cited as one of the possible good signs of organization. I see them as a clearest example of what you said when you have technological advance without social advance because basically we have to burn oil because there was no other way than the proof of work to distribute bitcoins. Do you think this is related to some more intrinsic characteristics of computer scientists that this belief not just in current democratic institutions but in democratic processes in general, which makes it difficult maybe to have even just discussions because if they are pointless, we will not have them? Yeah. So you're asking like the anonymity that a computer allows me or that the internet allows me, does it break down these types of conversations? Yeah. So it's an anarchic background compared to the idea that democracy can help us have social advances. Yeah. I mean, this is a massive debate and something that I thought about including, but I just didn't have time to include like for example, these people that believe that we should have parts of the internet where you have to be yourself, right? Allowing us to have, let's say like use the internet as a system of political or social trust, right? And then you also then have governments using that to actively persecute activists and using the same like, okay, I can publicly identify you or I can easily de-anonymize you amongst big data so I can actively persecute you. And this is this like thing, the more that we create systems that we can use for perhaps creating these types of like public conversations where I can represent myself and on a global stage and be an identifiable actor is also a problem. And this is nothing new, right? This is something that since way before the internet, but it means that perhaps, yeah, in some cases, anonymization and these types of things are necessary to protect people. But I think it's a really interesting debate and there was a few people that I wanted to include that are having this debate openly about e-democracy and so forth. And I think that it's fascinating and it's something that, yeah, hopefully we one day can trust, let's say, international institutions to help support people if they come forward publicly and they identify themselves and they speak of atrocities in their country or so forth, Snowden, that they perhaps then are supported globally and have protections globally. And I just don't know if we're there yet, unfortunately. Okay, last question. So my question was a bit based on what you said right now. How exactly do you come out after you find something that's just seriously unethical, exactly like Snowden? He discovered that NSA was really not doing what it was supposed to do and you got to a point where, hey, you're just getting everyone's data and you're doing something that's incredibly more. How exactly do you come out? I mean, you start to dispose information that's not really for the public, but if you keep on doing what you're doing, you're actually compromising more. It's a very moral shaky ground. But your recommendation or your view. Yeah, so I think the protections for whistleblowers within computing, within internet and these types of things, we have to really evolve the conversation. That being a whistleblower for something like a massive multinational corporation or a massive multinational state intelligence agency, like these are things, where do you then go? Like who's supposed to protect you, right? And I think that this is particularly dangerous within our field. If I worked for a massive multinational corporation and I knew that we were doing something wrong, what am I supposed to do? If I come forward, I will most likely be fired. There will probably be a target of everything against my name, make me out to be an idiot, an unethical human and so forth. And if we as a community have more of ways that we treat whistleblowers and more ways that we protect whistleblowers, maybe people would feel more comfortable, right? Because I also agree that what are you supposed to do? And in some of these stories, I'll be publishing a little bit more about this and I'll be putting some stories on my blog of some of the folks and what they did. But for example, there is a man called David Parnas and he was a computer scientist and worked in Canada for the US Department of Defense along with a few other multinational defense agencies. And he wrote a big public resignation letter and he sent it all around everywhere he could think of. This is the days of the mailing list, not particularly public internet. And he publicly said, this is unethical. Everything that's happening within this organization is not something I believe in. I refuse to work here and I don't know why my peers continue to do so. And I think that we can look at some of these references in history to perhaps think about how we can make some of the same statements, but that also we have to figure out how to protect one another when one another is willing to step forward and do that. Okay. Thanks again for this amazing keynote.
 All right, so now I want to welcome you, our next keynote speaker, it's Catherine  Jermul.  First time I saw Catherine was actually online on videos recorded at PyCon US where she was  talking about web scraping and data mangling and I really loved her talks.  Finally, we got to know each other actually in 2015 and as you know, like she's an author,  she wrote a book on data mangling on a Riley, she does video courses, she's one of the organizers  of PyData Berlin, she's a very active member of our community and now she's going to talk  about ethics and data...  Oh no, sorry, you're basically also like you're European basically, she was born in  LA but you're in Berlin since three years?  Yeah, about three and a half.  So and today she's going to talk about ethics and data, so this is not a technical keynote,  it's an ethics keynote, so I want to give you a very warm welcome to Catherine.  Thank you for being here, I know there was quite a fun social event last night, so I  appreciate you getting up early and joining us and I want to thank all of the organizing  committee for organizing such a great conference and also inviting me to be here.  Today I'm going to talk about a topic that is quite controversial in some circles.  Now ethics might be something that you learned about in school, it might be something that  you even debate on the world stage or when you talk about politics with your friends,  but it's not often that in the work that we do every day we get into ethical conversations.  But I wonder if it should be, I myself work as a data scientist and I myself face these  types of ethical conversations often when I'm building models or when I'm thinking about  how we use data and because of this I think that perhaps us even as a larger computing  field even if you don't work with data, perhaps this is something that we should be talking  about.  So what we'll do today is I'm going to take you through some of the history of computing  with an ethical focus.  So we're going to take a look at important moments in computing history and some of  the ethical reasoning behind them.  To begin with we will start with an IBM advertisement from 1960.  This is where Americans need to mind their game.  Around the clock radar electronic eyes watch those guys and report what they see to say  the vent system of the United States Air Force.  Here it is saying center on.  Oops sorry.  I don't know how to make this.  Let's go away.  All right.  We'll actually move forward because in the interest of time and in the interest of all  of your time what this is an advertisement for and what it goes on to explain is the  SAGE system and this system is a satellite air system and what it is supposedly doing  is tracking all of the aircraft over the United States and its goal is to have these people,  these intelligence workers to be able to click on a screen at these unknown aircrafts and  then fire missiles at them and destroy them midair.  Therefore protecting the United States from quote unquote Soviet attack.  This was a system that was designed by a lot of intelligent computer scientists.  A lot of people that were building IBM's best computers at the time in the late 50s and  early 60s.  And when I saw this advertisement I mean it's quite over the top.  I will have to post it later so you can look at it yourself.  But I wondered what the computer scientists who were working on this machine thought.  I wonder if perhaps they were at all concerned ethically about their creation.  And in testing actually this SAGE system was wholly ineffective.  In one test called Operation Sky Shield it would have only neutralized 25% of targets  leaving 75% of missiles incoming and to whatever devastation they were able to make.  And so I wonder if when they saw this advertisement and it's made to seem like ah you're completely  safe and secure because we have a computer and the computer will take care of it all.  I wonder what the computer scientists actually thought the computer engineers who were building  this system.  And I wonder if they were concerned with how it was marketed and if they were concerned  about failure.  So this is a constant problem in computing.  Computing has been touched by the military, has been touched by state intelligence systems  in a lot of ways throughout history.  The first computer scientist and mathematician we'll look at is Norbert Wiener.  And Norbert Wiener was a famous statistician in his time and he worked with neurologists  and other mathematicians to help discover some of the ways that our brain sends electricity.  So this is when we were first discovering how electrical signals travel through the brain  and when we were first discovering neural networks.  And he also did some statistics for the military.  In fact his work went on to contribute to the minimal mean squared estimator which essentially  allows us to estimate for he used it for flight paths for missile defense.  So he himself worked with the military, he himself probably had his own ethical qualms.  But his seminal work is a book called Cybernetics and he was actually one of the first to coin  this term.  This is this idea of using computers as an aid to help us make decisions.  And in Cybernetics he has a quote that I think is important to share today.  I'll read a slightly longer version than we have here.  I have said that this new development, computers, has unbounded possibilities for good and for  evil.  It gives the human race a new and most effective collection of mechanical slaves to perform  its labor.  Such mechanical labor has most of the economic properties of slave labor.  Although unlike slave labor it does not involve the direct and moralizing effects of human  cruelty.  However, any labor that accepts the conditions of competition with slave labor accepts the  conditions of slave labor and is essentially slave labor.  The key word of this statement is competition.  It may very well be a good thing for humanity to have the machine remove from it the need  of menial and disagreeable tasks.  Or it may not.  I do not know.  When I read this quote and I realized it was written in 1948 I was a bit surprised that  I don't feel like we've moved much further along in the conversation.  I feel like this quote could be shared in an article today, in a blog post today, in  any type of news today as we debate automation and jobs.  The people in our field we work to automate tasks away.  Sometimes we work on systems that will entirely replace an industry.  And therefore are we then responsible for the degrading of lives that happens when that  occurs?  Perhaps it's better that nobody has to be a factory worker or that nobody has to drive  a taxi if they don't want to or be a truck driver and so forth.  But what does it mean?  Is it the tyranny of the few?  Is it perhaps those of us that can automate away things and the societies that can afford  to automate away things versus those that perhaps can't?  And if those other societies or if other nations can't keep up does this mean that they're  essentially competing against these robots against this as he so-called slave labor?  The next mathematician that we'll look at and great computer scientist is Joseph Weisenbaum.  Now Weisenbaum was essentially a contemporary.  He was a German Jew and he was born in Berlin.  His family escaped Nazi persecution by emigrating to the United States and he grew up and was  educated primarily there.  He became a professor at MIT in the 1960s and you might know of his work because he  built Elizabot.  So for those of you familiar with natural language processing or perhaps you've just  heard about it, the Elizabot was probably the first chatbot and it used natural language  patterns to essentially mimic human speech and act as a therapist.  Now Weisenbaum was also famous in his own right for some other inventions that he did.  In fact, this is a photo of him at DeSite in 1965 and he's demonstrating remote login  to his MIT machine.  Weisenbaum came to become quite active politically within his time and he actively worked to  challenge those around him on these ethical and political concerns that he had.  I think he felt it especially important given his family's history to talk about ethics  in computing and what he saw around him.  And he began to question whether AI and computer science were a force of good in the world.  This is a quote from an interview in an MIT publication in 1985 where I feel like he was  particularly candid about his feelings about computing.  I'll again read a longer quote.  I think the computer has from the beginning been a fundamentally conservative force.  It has made possible the saving of institutions pretty much as they were, which otherwise  might have had to be changed.  For example, banking.  Superficially it looks as if banking has been revolutionized by the computer, but only  very superficially.  Consider that say 20, 25 years ago, the banks were faced with the fact that the population  was growing at a very rapid rate.  Many more checks would be written than before.  My response was to bring in the computer.  By the way, I helped design the first computer banking system in the United States for the  Bank of America 25 years ago.  Now if it had not been for the computer, if the computer had not been invented, what  would the banks have had to do?  They might have had to decentralize or they might have to regionalize in some way.  In other words, it might have been necessary to introduce a social invention, not just  a technical invention.  And this quote for me brought me pause as somebody that aims to make unmanageable tasks  manageable.  I aim to make data clean and accessible to use large and disparate data sets to make  inferences or determine some sort of meaning or signal.  Am I helping consolidate power?  By doing the job with maybe 30 lines and a scikit-learn import, by doing the job that  perhaps would be very untenable, take quite a long time, or maybe even impossible, am  I therefore helping consolidate this power?  Am I using technology to thwart social progress?  I don't know the answer to this, but these are the types of questions I'd love to hear  your feedback on.  Moving on throughout history, we have Oliohan Dal and Christian Nygaard.  They're essentially the fathers of object-oriented programming.  So we have them to think.  Simula 1 and Simula 67, the languages that they developed, were computer simulation languages.  They were used to simulate physics in a military laboratory.  And Christian Nygaard here on the right was a staunch leftist and activist throughout  his entire life.  In fact, he eventually identified as a socialist.  In his belief system, he of course believed in supporting workers' rights, and so when  he was approached by the Norwegian Iron and Metal Workers Union to help them build a system  so that their jobs would not be automated away and instead they would learn computing  skills, he jumped at the opportunity.  And he was able to work with them and build what many people think is the first example  of participatory design, one which the active participants or users or workers are able  to help make the design process and able to help determine what the system looks like.  And he essentially helped them build an operations research type of scheduler and backlog organizer  for the union.  He was also quite famous for walking away from military work.  In 1960, he quit his job at the defense research in Norway and took several of his teammates  with him.  He often joked that because of that, he had the most funding request rejections out of  any person in Norway.  He gave a speech at the Iris Conference, which is a group of Scandinavian researchers that  meet.  And I'll read a slightly longer quote from it.  You need a self-defense against yourself and the temptations to choose a comfortable but  wrong way out in critical situations.  But compromises may be necessary.  The greatest danger then is not the acceptance of a dubious compromise, but in not being  cynical and honest about it.  Your mental processes will try to justify your actions to yourself, making the compromise  the desired solution.  And you will change yourself if you're not honest and astute.  Bo Dalbom asked me to talk about the iron and metal project.  Why?  Many people don't know about it properly, he said.  And some have forgotten those aspects that ought to disturb them as their environment  pushes them slowly to the right.  Perhaps I should ask some questions to those in the audience who believe that they have  been influenced by the project and its successors.  Has anyone resented the content of your work recently?  If not, what is your excuse?  Have you had any real conflict in your research activities lately?  Or does such conflicts only belong to your now romanticized glorious radical past?  Will your recent research to any extent increase the power to influence their own fate for  people with whom you feel solidarity?  He went on to quip that he's joking a bit, but I think the message remains clear.  If we're building systems, tools, algorithms, and so forth, if we're building these and  they actively work against our politics, against our ethics, against our morality, however  it is you choose to make valuable decisions in your life.  If we do that, why are we doing that?  And if we choose instead to build tools and systems that we believe support our politics,  that we believe support our ethics, and we distribute them, can we be then a form of  justice?  Can we therefore spread instead of spreading, let's say, something we don't believe in,  can we therefore become a way to spread something that we do believe in?  Or as he suggests, are we just constantly perhaps making compromises, and do we need  to be very honest with ourselves about the compromises that we are making?  Andrei Urshok is a prominent figure in Soviet era computer science and cybernetics.  He worked on numerous inventions along the same time, working in parallel and often collaborating  with several computer scientists in the United States and the UK.  He's actually famous for likely writing the first optimizing compiler for a language that's  more complex than Fortran.  He wrote this for the alpha language that he worked on.  And he was a really big proponent of education.  In fact, he's probably the first computer scientist to talk about computer literacy.  He built several schools during his time, he appropriated funding for them, and this,  for example, is a photo at the USSR Academy of Sciences during a summer school for young  programmers.  His belief in teaching and in learning, and that being an important part of computer science,  is really espoused in his speech titled Aesthetics and the Human Factor in Programming, which  he gave at a 1972 computer conference in the United States.  I'll read the full quote of which only a part is shown here.  In past ages, the ability to read and write was considered a rare, God-given talent gift.  The destiny of a limited group of the specially chosen.  In the present age of general literacy, we perceive reading to be a universally attainable  accomplishment, but we are tempted to single out a new elite group who become arbiters  between the lay generality of mankind and the arcane informational model of the world  hidden in the machine.  Is it not, however, the highest aesthetic idea of our profession to make the art of  programming public property, and thereby to submerge our elite exclusiveness within a  mature mankind?  Indeed, I feel like Urshav approaches these ethical problems slightly differently than  some of his Western contemporaries.  His idea is not, do we hold some power that we should then further hone for good or for  evil.  His point is, should we even hold this power at all?  Should everyone learn to code?  Should everyone learn how a computer works literally on the inside, not just how to turn  it on and off?  And if not, does our own understanding of computers, does our own ability to code, does,  for example, the data scientists in the room, our understanding of statistics and machine  learning models, does this give us some special power or privilege?  And if it does give us a power or privilege, are we an elite?  And what is being part of an elite mean?  Does it mean we have a responsibility?  Does it mean that we have to do things differently?  Moving on to the networking era, which is always fun, right?  We all love the Internet.  Here you can see the employees at BBN, which is Bolt, Baranik and Newman, a Cambridge research  firm.  And BBN won a contract with the Advanced Research Projects Agency, otherwise known as ARPA.  And their project was to build the first computer network in the United States.  They're seen here in 1969 along with their interface messaging processor machine, which  is essentially the first router.  And in case you don't know, this is the same group that started to create the protocols  and the standards that we use in today's Internet.  There's a bunch of amazing engineers and architects that I could talk about in this photo, but  I'm going to focus on two of them.  Here in blue, I have several, and in yellow, I have Bob.  Why just talk about the two of them?  Well, first and foremost, because they went on to have longer careers and do quite a lot  of work after ARPA.  But secondly, because they were very outspoken in their political and ethical beliefs, and  they went in quite different directions.  So I think it's perhaps a good case study to explore the life of a computer scientist.  We'll focus on Bobcon first.  Bobcon worked on networks and essentially networking protocols.  He along with Vincent Surf was able to create the TCP and IP protocols and standards, which  are mainly unchanged today, which is pretty amazing, feet.  He's essentially a networking genius.  He went on to build very large networks for the US military at DARPA, and then after that  he went to IPTO, which is the Information Processing Techniques Office.  Then he heard about this new project brewing in Japan called the Fifth Generation Computer  Project.  And this idea was how can we create true AI thinking machines?  How can we create these decision support systems that we might be able to use?  At that time, he then pitched the idea of the strategic computing initiative, of which  this is one of the plans.  So for the strategic computing initiative, you have, as you can see at the bottom, all  of the infrastructure.  These are the networks that Bobcon loved.  This is something that he loves to work on.  He was able to get quite a lot of funding to just build networks and come up with new  networking concepts.  On top of that, they would then try to build some chips, some different hardware designs,  and eventually working to natural language processing and speech and vision recognition  and so forth.  On top of those technologies, they would build autonomous systems, pilots associates, and  battle management.  And this was all to, and I quote, develop a broad base of machine intelligence technology  to increase our national security and economic strength.  So what does this mean?  I mean, when I first saw this diagram, I said, well, perhaps Bobcon didn't even know about  how his networks would be used.  Maybe ethically for him, he just really wanted to work on networks.  I can understand and appreciate that.  And then I found a quote from around this era of Bobcon.  And it states, the nation that dominates this information processing field will possess  the keys to world leadership in the 21st century.  As I began to read more of his writing during the time and writings of his peers at SCI,  for example, Lynn Conway, who is a prominent computer scientist in her own right, it was  clear that they were very well aware of how their programming would be used.  In fact, it's rumored that the chart that we saw in the last slide, that Bobcon himself  designed that pyramid.  So it's clear that he knew he was building weapons.  What I wonder is, what does it mean when we have to take military funding or let's say  corporate funding of corporations, maybe we don't agree with, to do our work?  Does this make us ethically culpable in what the greater process is?  And if it does make us ethically culpable, if something goes wrong, let's say civilians  are bombed and so forth, does that make you as the computer scientist or as the data scientist  or the machine learning expert, does it also mean that you are culpable?  And if you aren't culpable, then who is?  Going back to Bobcon's peer, Severo Ornstein, he was a co-worker again and a peer at ARPANET.  And here we see him with Laura Golt, who was a computer scientist and activist as well.  Severo's work at ARPANET was as lead hardware expert.  So he built most of the hardware for the first router and also did some of the software programming.  He later went on to join Xerox PARC and he worked on the Dorado, which was the fastest  processor at the time in operating system.  So around the same time that Bobcon was petitioning the government for millions of dollars to build  weapons, Severo Ornstein and Laura Golt were starting another initiative.  And this was for a group called Computer Professionals for Social Responsibility.  This is one of the patches that the Computer Professionals for Social Responsibility gave  out at a joint AI conference, most likely the one in Los Angeles in 1985.  And some older folks in the room or perhaps some that knew a little bit of Americana history.  This is a play on a famous Cold War era advertisement, if you will.  It was a public service announcement that came on and it would say, it's 11 p.m.  Do you know where your children are?  And its goal was to enforce curfew and to make sure that all the young people were inside  at night in case of danger.  Now this play on it is it's 11 p.m.  Do you know what your expert system just inferred?  And we see the telltale mushroom cloud of an atomic bomb.  And these are the same expert systems that Bobcon himself was trying to build.  I mean, they never really came too much, right?  At that time it was very quickly before what we all know now as AI winter.  And these expert systems were never much more than large statistical models.  But the fear at the time was that they would be treated as experts.  It was that if the system itself said, okay, we're under attack, it's time to launch the  nukes, that that is something that would actually happen and that a human would go about doing  it based on this expert system.  That was a massive concern for CPSR, the computer professionals for social responsibility.  Now CPSR was around actually until 2013.  And they gave out fairly regular awards actually named Nelvarth Vina and highlighting people  in the field who are working on social responsibility issues as computer professionals.  It's a pretty amazing list.  We've actually covered some of the people on the list, but it goes on to talk about whistleblowers  and other people who really worked actively to act as an ethical agent.  But I think what we can really take away from CPSR or what I took away from CPSR is that  it was starting this type of activist conversation amongst computer professionals.  And it was asking, what can we do as people who are concerned about the world, that as  people who are concerned about social responsibility, how can we help?  How can we be a part of change or a part of something that we believe brings justice to  the world?  And I think that it's quite sad when you read about the history of CPSR that around the time  of the personal computing revolution in the mid to late 80s and early 90s, they saw their  numbers dwindle.  And I think one of the problems of this is computing went from being a social experience  in the computer lab, working with their peers, chatting probably a lot about these types of  issues and theories.  It became a personal and a lone experience.  This was also the time that is often tracked to the gender disparity within our field.  That's often tracked that the advertisements were all of a sudden, be a gamer, it's really  cool for boys to have computers.  And so I feel like this was perhaps not intentionally, but unfortunately, one of the fatalities of  that era was this conversation about computer professionals for social responsibility.  But CPSR was not the only ones fighting for more professionalism in the field or fighting  for us to be having these conversations.  You might know of Karen Spock Jones.  She's quite famous.  She's a statistical, she's a statistician as well as a natural language processing expert.  And she worked at the Cambridge Language Research Unit on a variety of NLP research.  Probably her first forays, I guess, are into looking at inferring meaning from the Sari  definitions in text.  You might have also heard of TFIDF, term frequency, inverse document frequency.  It's a popular way to extract important words or phrases from a text when you have a larger  corpus.  She wrote the paper that defined the IDF portion of the equation.  So she's pretty important when it comes to natural language processing.  But that's not her only contribution.  Spock Jones was an outspoken critic of the gender disparities within our field.  She also was quite a critic of not having some sort of professional code.  She thought that the growing importance of computers in people's lives meant that we  had more and more influence and she was concerned that we didn't have any type of licensing  board review, that we didn't necessarily have any standards for our own code, our own actions.  I'll again read a slightly longer quote.  I certainly think that professionalism is very important.  To be a proper professional, you need to think about the context and motivation and  justifications of what you're doing.  You don't need a fundamental philosophical discussion every time you put your finger  on the keyboard.  But as computing is spreading so far into people's lives, you need to think about these things.  I've always felt that once you see how important computing is for life, you can't just leave  it as a blank box and assume that somebody reasonably competent and relatively benign  will do something right with it.  For a quote, again, evokes this idea, who is responsible?  If I build a program or write a script or build a model that does something unethical  or that can be used unethically, am I responsible for when somebody does that?  If I'm not responsible, who is?  As especially machine learning and numerous other fields that we work in as they touch  more and more lives, as they're used even by legal experts or by the police or by state  intelligence or by doctors, are we not also beholden to these same licenses, these same  ethics that these fields have?  Now, I'm not necessarily saying we need to have a licensing or we need to have a particular  code, but if we have no standards, if we have no way to review one another's work and to  think about whether we're falling in ethical principles or whether we're helping or hurting  lives, then we don't even have a starting place for these conversations on professionalism.  And if we're not willing to take on the burden of thinking about these things, then does the  burden actually go away or are we just ignoring it?  Finally, we've reached some of the outspoken leaders of today's AI in today's era.  This is Joanna Bryson.  She is a professor of computer science who speaks on AI and natural intelligence at the  University of Bath.  And she's been working on the field of AI ethics since the 90s.  She is a prolific writer and she's published all the time.  She has a paper over this past year on the epics of looking at word vectors and different  gender disparity within them.  But she also has some older works.  For example, her paper, Just Another Artifact, Ethics and the Empirical Experience of AI,  articulates this problem of humans over identifying with artificial intelligence.  She essentially says that instead of treating it like a book or a reference, we treat AI  as a human actor, as another intelligent human being with reason and common sense and so  forth.  And she states that this is quite dangerous.  If we start treating AI as more expert than ourselves, if we start treating it as the  sole singular expert in the room, then what happens when it makes a mistake?  So she's been quite outspoken on these issues.  On one of her longer posts, she articulates some of her well thought out opinions on how  we use data.  I'll read again from a longer quote.  As we in the computational social sciences learn more and more, our models of human behavior  get better and better.  As our models improve, we need less and less data about any particular individual to predict  what they're going to do.  So just practicing good data hygiene is not enough, even if that was a skill that we could  teach everyone.  My professional opinion is that there's no going back on this.  But that isn't to say that society is doomed.  Think of it this way.  We all know that the police, the military, even most of our neighbors could get into  our house if they wanted to, but we don't expect them to do that.  And generally speaking, if anyone does get into our house, we are able to prosecute them  legally and claim any damages back from insurance.  I think our personal data should be like our houses.  First of all, we shouldn't be seen as selling our own data, just leasing it for a particular  purpose.  This is the model software companies are to use for their products.  We should just apply the same legal reasoning to we humans.  Then if we have any reason to suspect our data has been used in a way we didn't approve,  we should be able to prosecute.  That is, the applications of our data should be subject to regulations that protect ordinary  citizens from the intrusions of governments, corporations, and even friends.  What Joanna Bryson is asking of us here is to actually be better than the current laws.  Now we could get into some debates about the new EU privacy regulations and the fact that  yes indeed it has some ways that I can control my own personal data and I can request information  about how my data is used.  But those are clearly not global and they have yet to be determined in the courts how  they'll be enforced.  What she's asking us to do is to treat people's data like their property.  Would you buy and sell another person's property without them knowing it?  And if not, why do we do this with customer data?  Would you collect pieces of another person's property without properly informing them in  plain English so that they can understand what they're agreeing to?  And if not, why do we do it in our terms and conditions?  And a particular concern of mine regarding security, would you take your friend's laptop  or computer or books or whatever it is that they have and would you leave it on a public  table outside for everybody to walk by and perhaps look at?  And if not, then why are you leaving your databases with protected information or your  models with protected information on the public internet?  Debating ethics at work is nothing new.  My great-grandmother was a pretty amazing woman.  Here's her.  Her name was Adele Matilda Rich.  Her on the left is her growing up in New York City and she used to tell me really fun stories  in the 20s and 30s of growing up in New York City.  And her on the right is her with a very young version of me.  Now, Adele, as I used to call her, was amazing.  She has a lot of good memories that I have in my mind.  But the reason why I bring her up is the fact that she worked as a secretary on the Manhattan  project.  The Manhattan project was the project in case you don't know the code words for the nuclear  bomb in the United States.  And she worked as a secretary.  She used to tell me fun stories that she did something important for the war and that she  had to burn her notebooks and so forth.  And I just thought it was a fun story until I got older.  She died when I was a child, so I was never able to have these types of conversations  with her.  But I did talk with my mother, who I know had several conversations.  She said that Adele at first didn't know exactly what she was working on, but slowly put the  pieces together over time and then of course when the bombs were dropped, she was quite  aware.  And I asked my mom, did she ever regret it?  Did she ever stay up at night wondering if what she did was right or wrong?  And my mom said that she thought that Adele felt like many of her generation that she  was helping the good guys and that any deaths that were created by the atomic bombs dropped  were just small casualties in a larger war.  I wonder if I could go back and ask her if we could have a longer conversation about  this.  But it also makes me wonder if my own privilege now, something that she worked so hard to  do.  She worked to help make sure our family could survive through the war and the Great Depression.  And she made it so that my mom could be the first one to go to college.  And my mom going to college meant that I was able to get a computer and that I was able  to go pursue computer science.  So maybe it is wrong of me to question something like that.  Perhaps she was not in a position to make a decision between ethics and feeding her family.  And perhaps it is rather ignorant of me to even ask that question.  But I also wonder if it is not a right for all of us to have rewarding work that's both  challenging mentally, financially rewarding, and also ethically something that we can support.  So I challenge us as a community.  As data scientists, web developers, if you work on MicroPython, CorePython, Jython, whatever  it is that you do, Internet of Things, I challenge us to start having these possibly more difficult  conversations.  I challenge us to start thinking about who is responsible and how we can hold one another  accountable.  Now, you might not work on anything that touches ethics, but perhaps you do.  Or perhaps a coworker does, or a friend, or a colleague, or somebody you meet at this  conference, perhaps a future you, or a past you.  I know in my career I've been definitely asked to do unethical things.  And I've been asked to lie, to make charts look different, to make data look different  for the greater good, right, for extra funding, for whatever it is that the goal of the company  or the product is.  So I've had to have some of my own ethical decisions.  I think that hopefully what we've heard today and what we've shared today can start to be  a ground for a communal conversation.  I know that in a community like this one, we have values.  For example, I feel like as a Python community, we value diversity.  I feel that we value free software and open source, and I'm very proud of that.  I also feel welcome and supported today, just the same as my first Python conference in  2010.  So we have these shared beliefs and these shared values already.  But perhaps we don't have a larger understanding of how they relate to ethics in computing  as a whole, or how they could inform a communal ethic, or communal principles, codes of code,  and conduct, ideas that we share with one another, and conversations that we have openly.  Now it may not be that we come to some communal agreement, but just the idea of sharing our  stories openly, sharing stories like these from around the world openly that tell us  a little bit about the history, the ideas, and these deep questions and dilemmas that  we have.  Perhaps that is a way to start being a better force in the world.  We data scientists, programmers, Pythonistas, computer and software engineers, we do have  this cultural and ethical history.  I hope that it's shown this is only just a few of what I was able to find in my research.  I want us to share these stories, and I want to hear your stories and have you share your  stories as well of these ethical dilemmas that we face and the decisions that we make  around them.  And I truly, truly believe that in doing so, we are more.  That when we have these conversations together, we are actually contributing to a greater  good.  Perhaps by having all of these conversations and working together, we can make sure that  AI, machine learning, natural language processing, and Python have a bright future in justice.  And that that's something that we can be really proud of.  And I know that I want to be a part of that, and I hope that you do too.  Thank you so much for being here, and I hope to continue this conversation throughout the  conference.  Thank you very much, Capric.  I think it was an amazing keynote.  Now everybody is really quiet because there's a lot of thinking going on in the room, but  we have time for questions.  So who has questions or comments?  And if you're able, it would be nice if you could talk about that.  Thank you very much.  I have a question about, we'll just say, banks.  There was no computer that they could call different.  But thanks to the computers, we have automated ways right now, like PayPal, and the cryptocurrencies,  which basically allows us to operate without banks.  So what do you think about this?  Yeah, so I definitely think I shared some folks today throughout history, and there  was a few that touched a little bit on this, but I feel like this is definitely something  more of our generation, our era, the idea of using kind of some of this technology to  subvert the norm, right, to actually actively work against corporate influence on our life  or government influence on our lives.  So I think that there's a lot of ways that technology can be used to kind of actively  work against some of these ideas.  I completely agree, and I think that that's a really important conversation that unfortunately  didn't quite fit in in this talk, but I think that's really important to share and think  about.  And this is kind of this idea of us creating technologies that actively support what we  believe in, and then as we spread them in the world, that that hopefully makes more things  possible.  Like, I look at teams like the Signal team and what they're working on, and I think it's  really great.  There's some amazing work out there, and perhaps you work on those things as well, and these  are really interesting conversations to me and things that we should also be highlighting,  right?  Thank you.  Hi, thank you for the amazing talk.  I would like to ask you a question about auto-automatization, especially if you see difference between  what will happen probably with this machine learning revolution in some sense, and something  like the industrial revolution with what happened in China, because the advantage that  I see, or those other things that happened in the past, is that in fact they lifted a  lot of people from poverty, even though it was not a great process in some sense, that  was the main advantage.  One should always say when you talk about, you know, auto-automatization machine, it  has also human side, which was helping people.  So do you think there is some differences between what will happen and what has already  happened and why this should raise some different question regarding ethics?  So, Zai, the question exactly is about how the computer revolution is different than  revolutions before it, or, sorry.  And the question is how do you think ethically this is different in some sense, one should  be what we should think about?  Is this just automatization on a larger scale, or is something, in a different way?  Yeah, I wonder about that, because, yeah, clearly the industrial revolution was, and  other previous revolutions, for example, the printing press and so forth created their  own ethics and their own issues.  What I wonder about with computers is, like as we've seen with Moore's Law, and then  also with the new post-AI winter, we're seeing quite a lot of progress in a short amount of  time.  I don't know if that's the same amount of progress.  I obviously wasn't alive then, and I'm no expert on previous revolutions.  I'm not even an expert on this revolution.  So I don't exactly know, but I wonder if the pace and perhaps the expanse is slightly more  impactful now, that maybe because of the internet and because of the advances that we've made  in computing, perhaps we have a more quick impact on people's lives.  I don't know if this is true, but sometimes I think that perhaps this might be the case,  but I would be curious to hear more of your thoughts on that topic as well.  Great talk, really great.  I was thinking about the ethical side of what we do on a daily basis is really important,  but my concern is more, for example, the people who started investigating the outcomes and  their subparticles many, many years ago, they had no idea that this will lead to the Manhattan  Project.  So do you have any advice on how we can try to predict the future implications of our  work and how they can, the ethical result 50 or 100 years from now?  This is something where if I had an answer for you, I would be telling it to the world  from the rooftops.  And I don't know, I think perhaps we need to, I echo that concern.  I don't know what things that I work on today and things that people much brighter than  me work on today.  I have no idea what large impact these will have on the world a decade from now, even  five or 10 decades from now.  But I do think we should be thinking about it, and I think that I was heartened to when  I was reading and doing research for this, that some of these people were trying to project  and think about what this could do in the future.  Now a lot of the times they were pretty far off.  But I think that it doesn't hurt to start the conversation now and to work together  to perhaps see what this might mean in 10, 50 years.  And for example, how we can protect against things that we're fearful of and also support  ideas that we think would be really great things that we can do with technology.  Thank you for an awesome keynote.  I think at least from my point of view, it's pretty obvious that automation of work is  a trend that will continue and that will probably worsen any social disparities that we have  now.  Do you have any ideas where to start a conversation about how technologists can help in this situation  instead of worsen the state that we are already in?  Yeah, so I think that as I was thinking about this and when I found the computer professionals  for social responsibility, now there are some groups that are working on these types of  things.  There's like fairness, accountability, and transparency, machine learning group and  conference.  There are some other ones that are trying to start these conversations about how we can  be socially conscious actors when we do automation.  I think that this would be a great thing to have a group around or a conference around  or ideas around.  I am no expert.  I am just a person like you who is concerned.  And I think that the more that we are able to share our ideas, I think that honestly,  collectively, we are much more smart than any of us individually.  And perhaps by having these types of debates openly, we can figure out maybe thoughtful  ways that we can automate so that perhaps the impact is lessened.  I also agree automation is definitely not going to go anywhere and nobody is going to  stop automating because of a keynote or because somebody has concerns.  So I think that, yeah, having these conversations and collectively coming up with some good  solutions or some good ideas is most likely the most powerful thing that we can do.  We have three more questions.  And if you promise to keep them short, I'm going to take them.  But I think probably since it's such an interesting subject, would you be willing to do an open  space for that?  Yeah, sure.  Sure.  So basically, if you don't know open spaces yet, we have a large room there.  You can just join there and make up your own EuroPython.  And there's a, yeah, that's what it's for.  So there's a whiteboard downstairs.  You can use it.  So we're going to put an open space for this on.  I'm going to tweet all also about it because I think many are interested and there's like  three more short questions.  Yeah.  Yeah, cryptocurrencies have been just cited as one of the possible good signs of organization.  I see them as a clearest example of what you said when you have technological advance  without social advance because basically we have to burn oil because there was no other  way than the proof of work to distribute bitcoins.  Do you think this is related to some more intrinsic characteristics of computer scientists  that this belief not just in current democratic institutions but in democratic processes in  general, which makes it difficult maybe to have even just discussions because if they  are pointless, we will not have them?  Yeah.  So you're asking like the anonymity that a computer allows me or that the internet allows  me, does it break down these types of conversations?  Yeah.  So it's an anarchic background compared to the idea that democracy can help us have social  advances.  Yeah.  I mean, this is a massive debate and something that I thought about including, but I just  didn't have time to include like for example, these people that believe that we should have  parts of the internet where you have to be yourself, right?  Allowing us to have, let's say like use the internet as a system of political or social  trust, right?  And then you also then have governments using that to actively persecute activists and using  the same like, okay, I can publicly identify you or I can easily de-anonymize you amongst  big data so I can actively persecute you.  And this is this like thing, the more that we create systems that we can use for perhaps  creating these types of like public conversations where I can represent myself and on a global  stage and be an identifiable actor is also a problem.  And this is nothing new, right?  This is something that since way before the internet, but it means that perhaps, yeah,  in some cases, anonymization and these types of things are necessary to protect people.  But I think it's a really interesting debate and there was a few people that I wanted to  include that are having this debate openly about e-democracy and so forth.  And I think that it's fascinating and it's something that, yeah, hopefully we one day  can trust, let's say, international institutions to help support people if they come forward  publicly and they identify themselves and they speak of atrocities in their country or  so forth, Snowden, that they perhaps then are supported globally and have protections  globally.  And I just don't know if we're there yet, unfortunately.  Okay, last question.  So my question was a bit based on what you said right now.  How exactly do you come out after you find something that's just seriously unethical,  exactly like Snowden?  He discovered that NSA was really not doing what it was supposed to do and you got to  a point where, hey, you're just getting everyone's data and you're doing something that's incredibly  more.  How exactly do you come out?  I mean, you start to dispose information that's not really for the public, but if you keep  on doing what you're doing, you're actually compromising more.  It's a very moral shaky ground.  But your recommendation or your view.  Yeah, so I think the protections for whistleblowers within computing, within internet and these  types of things, we have to really evolve the conversation.  That being a whistleblower for something like a massive multinational corporation or a massive  multinational state intelligence agency, like these are things, where do you then go?  Like who's supposed to protect you, right?  And I think that this is particularly dangerous within our field.  If I worked for a massive multinational corporation and I knew that we were doing something wrong,  what am I supposed to do?  If I come forward, I will most likely be fired.  There will probably be a target of everything against my name, make me out to be an idiot,  an unethical human and so forth.  And if we as a community have more of ways that we treat whistleblowers and more ways  that we protect whistleblowers, maybe people would feel more comfortable, right?  Because I also agree that what are you supposed to do?  And in some of these stories, I'll be publishing a little bit more about this and I'll be putting  some stories on my blog of some of the folks and what they did.  But for example, there is a man called David Parnas and he was a computer scientist and  worked in Canada for the US Department of Defense along with a few other multinational  defense agencies.  And he wrote a big public resignation letter and he sent it all around everywhere he could  think of.  This is the days of the mailing list, not particularly public internet.  And he publicly said, this is unethical.  Everything that's happening within this organization is not something I believe in.  I refuse to work here and I don't know why my peers continue to do so.  And I think that we can look at some of these references in history to perhaps think about  how we can make some of the same statements, but that also we have to figure out how to  protect one another when one another is willing to step forward and do that.  Okay.  Thanks again for this amazing keynote.
Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_0_59.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_60_131.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_132_179.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_180_227.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_228_311.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_312_479.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_480_539.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_540_671.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_672_791.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_792_839.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_840_971.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_972_1151.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1152_1223.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1224_1271.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1272_1331.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1332_1559.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1560_1619.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1620_1691.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1692_1751.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1752_1859.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1860_1931.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_1932_2147.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_2148_2303.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_2304_3647.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_3648_3827.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_3828_3995.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_3996_4199.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_4200_4355.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_4356_4403.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_4404_4595.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_4596_5075.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_5076_5291.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_5292_5447.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_5448_5519.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_5520_5651.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_5652_5987.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_5988_7787.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_7788_8351.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_8352_30551.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_30552_35903.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_35904_39275.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_39276_52091.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_52092_66755.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_66756_66839.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_66840_67139.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67140_67235.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67236_67295.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67296_67367.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67368_67415.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67416_67487.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67488_67655.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_67656_69431.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_69432_73079.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_73080_73199.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_73200_74555.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_74556_75395.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_75396_75995.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_75996_76379.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_76380_76583.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_76584_77195.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_77196_77267.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_77268_77579.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_77580_78851.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_78852_79043.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79044_79247.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79248_79331.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79332_79631.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79632_79691.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79692_79847.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79848_79907.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_79908_80027.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_80028_80099.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_80100_80783.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_80784_81503.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_81504_83603.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_83604_84671.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_84672_84839.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_84840_85655.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_85656_86771.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_86772_87479.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_87480_87527.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_87528_87575.png Katharine_Jarmul_-_If_Ethics_is_not_None_FtRbAePXUoI_87576_87676.png
All right, so now I want to welcome you, our next keynote speaker, it's Catherine Jermul. First time I saw Catherine was actually online on videos recorded at PyCon US where she was talking about web scraping and data mangling and I really loved her talks. Finally, we got to know each other actually in 2015 and as you know, like she's an author, she wrote a book on data mangling on a Riley, she does video courses, she's one of the organizers of PyData Berlin, she's a very active member of our community and now she's going to talk about ethics and data... Oh no, sorry, you're basically also like you're European basically, she was born in LA but you're in Berlin since three years? Yeah, about three and a half. So and today she's going to talk about ethics and data, so this is not a technical keynote, it's an ethics keynote, so I want to give you a very warm welcome to Catherine. Thank you for being here, I know there was quite a fun social event last night, so I appreciate you getting up early and joining us and I want to thank all of the organizing committee for organizing such a great conference and also inviting me to be here. Today I'm going to talk about a topic that is quite controversial in some circles. Now ethics might be something that you learned about in school, it might be something that you even debate on the world stage or when you talk about politics with your friends, but it's not often that in the work that we do every day we get into ethical conversations. But I wonder if it should be, I myself work as a data scientist and I myself face these types of ethical conversations often when I'm building models or when I'm thinking about how we use data and because of this I think that perhaps us even as a larger computing field even if you don't work with data, perhaps this is something that we should be talking about. So what we'll do today is I'm going to take you through some of the history of computing with an ethical focus. So we're going to take a look at important moments in computing history and some of the ethical reasoning behind them. To begin with we will start with an IBM advertisement from 1960. This is where Americans need to mind their game. Around the clock radar electronic eyes watch those guys and report what they see to say the vent system of the United States Air Force. Here it is saying center on. Oops sorry. I don't know how to make this. Let's go away. All right. We'll actually move forward because in the interest of time and in the interest of all of your time what this is an advertisement for and what it goes on to explain is the SAGE system and this system is a satellite air system and what it is supposedly doing is tracking all of the aircraft over the United States and its goal is to have these people, these intelligence workers to be able to click on a screen at these unknown aircrafts and then fire missiles at them and destroy them midair. Therefore protecting the United States from quote unquote Soviet attack. This was a system that was designed by a lot of intelligent computer scientists. A lot of people that were building IBM's best computers at the time in the late 50s and early 60s. And when I saw this advertisement I mean it's quite over the top. I will have to post it later so you can look at it yourself. But I wondered what the computer scientists who were working on this machine thought. I wonder if perhaps they were at all concerned ethically about their creation. And in testing actually this SAGE system was wholly ineffective. In one test called Operation Sky Shield it would have only neutralized 25% of targets leaving 75% of missiles incoming and to whatever devastation they were able to make. And so I wonder if when they saw this advertisement and it's made to seem like ah you're completely safe and secure because we have a computer and the computer will take care of it all. I wonder what the computer scientists actually thought the computer engineers who were building this system. And I wonder if they were concerned with how it was marketed and if they were concerned about failure. So this is a constant problem in computing. Computing has been touched by the military, has been touched by state intelligence systems in a lot of ways throughout history. The first computer scientist and mathematician we'll look at is Norbert Wiener. And Norbert Wiener was a famous statistician in his time and he worked with neurologists and other mathematicians to help discover some of the ways that our brain sends electricity. So this is when we were first discovering how electrical signals travel through the brain and when we were first discovering neural networks. And he also did some statistics for the military. In fact his work went on to contribute to the minimal mean squared estimator which essentially allows us to estimate for he used it for flight paths for missile defense. So he himself worked with the military, he himself probably had his own ethical qualms. But his seminal work is a book called Cybernetics and he was actually one of the first to coin this term. This is this idea of using computers as an aid to help us make decisions. And in Cybernetics he has a quote that I think is important to share today. I'll read a slightly longer version than we have here. I have said that this new development, computers, has unbounded possibilities for good and for evil. It gives the human race a new and most effective collection of mechanical slaves to perform its labor. Such mechanical labor has most of the economic properties of slave labor. Although unlike slave labor it does not involve the direct and moralizing effects of human cruelty. However, any labor that accepts the conditions of competition with slave labor accepts the conditions of slave labor and is essentially slave labor. The key word of this statement is competition. It may very well be a good thing for humanity to have the machine remove from it the need of menial and disagreeable tasks. Or it may not. I do not know. When I read this quote and I realized it was written in 1948 I was a bit surprised that I don't feel like we've moved much further along in the conversation. I feel like this quote could be shared in an article today, in a blog post today, in any type of news today as we debate automation and jobs. The people in our field we work to automate tasks away. Sometimes we work on systems that will entirely replace an industry. And therefore are we then responsible for the degrading of lives that happens when that occurs? Perhaps it's better that nobody has to be a factory worker or that nobody has to drive a taxi if they don't want to or be a truck driver and so forth. But what does it mean? Is it the tyranny of the few? Is it perhaps those of us that can automate away things and the societies that can afford to automate away things versus those that perhaps can't? And if those other societies or if other nations can't keep up does this mean that they're essentially competing against these robots against this as he so-called slave labor? The next mathematician that we'll look at and great computer scientist is Joseph Weisenbaum. Now Weisenbaum was essentially a contemporary. He was a German Jew and he was born in Berlin. His family escaped Nazi persecution by emigrating to the United States and he grew up and was educated primarily there. He became a professor at MIT in the 1960s and you might know of his work because he built Elizabot. So for those of you familiar with natural language processing or perhaps you've just heard about it, the Elizabot was probably the first chatbot and it used natural language patterns to essentially mimic human speech and act as a therapist. Now Weisenbaum was also famous in his own right for some other inventions that he did. In fact, this is a photo of him at DeSite in 1965 and he's demonstrating remote login to his MIT machine. Weisenbaum came to become quite active politically within his time and he actively worked to challenge those around him on these ethical and political concerns that he had. I think he felt it especially important given his family's history to talk about ethics in computing and what he saw around him. And he began to question whether AI and computer science were a force of good in the world. This is a quote from an interview in an MIT publication in 1985 where I feel like he was particularly candid about his feelings about computing. I'll again read a longer quote. I think the computer has from the beginning been a fundamentally conservative force. It has made possible the saving of institutions pretty much as they were, which otherwise might have had to be changed. For example, banking. Superficially it looks as if banking has been revolutionized by the computer, but only very superficially. Consider that say 20, 25 years ago, the banks were faced with the fact that the population was growing at a very rapid rate. Many more checks would be written than before. My response was to bring in the computer. By the way, I helped design the first computer banking system in the United States for the Bank of America 25 years ago. Now if it had not been for the computer, if the computer had not been invented, what would the banks have had to do? They might have had to decentralize or they might have to regionalize in some way. In other words, it might have been necessary to introduce a social invention, not just a technical invention. And this quote for me brought me pause as somebody that aims to make unmanageable tasks manageable. I aim to make data clean and accessible to use large and disparate data sets to make inferences or determine some sort of meaning or signal. Am I helping consolidate power? By doing the job with maybe 30 lines and a scikit-learn import, by doing the job that perhaps would be very untenable, take quite a long time, or maybe even impossible, am I therefore helping consolidate this power? Am I using technology to thwart social progress? I don't know the answer to this, but these are the types of questions I'd love to hear your feedback on. Moving on throughout history, we have Oliohan Dal and Christian Nygaard. They're essentially the fathers of object-oriented programming. So we have them to think. Simula 1 and Simula 67, the languages that they developed, were computer simulation languages. They were used to simulate physics in a military laboratory. And Christian Nygaard here on the right was a staunch leftist and activist throughout his entire life. In fact, he eventually identified as a socialist. In his belief system, he of course believed in supporting workers' rights, and so when he was approached by the Norwegian Iron and Metal Workers Union to help them build a system so that their jobs would not be automated away and instead they would learn computing skills, he jumped at the opportunity. And he was able to work with them and build what many people think is the first example of participatory design, one which the active participants or users or workers are able to help make the design process and able to help determine what the system looks like. And he essentially helped them build an operations research type of scheduler and backlog organizer for the union. He was also quite famous for walking away from military work. In 1960, he quit his job at the defense research in Norway and took several of his teammates with him. He often joked that because of that, he had the most funding request rejections out of any person in Norway. He gave a speech at the Iris Conference, which is a group of Scandinavian researchers that meet. And I'll read a slightly longer quote from it. You need a self-defense against yourself and the temptations to choose a comfortable but wrong way out in critical situations. But compromises may be necessary. The greatest danger then is not the acceptance of a dubious compromise, but in not being cynical and honest about it. Your mental processes will try to justify your actions to yourself, making the compromise the desired solution. And you will change yourself if you're not honest and astute. Bo Dalbom asked me to talk about the iron and metal project. Why? Many people don't know about it properly, he said. And some have forgotten those aspects that ought to disturb them as their environment pushes them slowly to the right. Perhaps I should ask some questions to those in the audience who believe that they have been influenced by the project and its successors. Has anyone resented the content of your work recently? If not, what is your excuse? Have you had any real conflict in your research activities lately? Or does such conflicts only belong to your now romanticized glorious radical past? Will your recent research to any extent increase the power to influence their own fate for people with whom you feel solidarity? He went on to quip that he's joking a bit, but I think the message remains clear. If we're building systems, tools, algorithms, and so forth, if we're building these and they actively work against our politics, against our ethics, against our morality, however it is you choose to make valuable decisions in your life. If we do that, why are we doing that? And if we choose instead to build tools and systems that we believe support our politics, that we believe support our ethics, and we distribute them, can we be then a form of justice? Can we therefore spread instead of spreading, let's say, something we don't believe in, can we therefore become a way to spread something that we do believe in? Or as he suggests, are we just constantly perhaps making compromises, and do we need to be very honest with ourselves about the compromises that we are making? Andrei Urshok is a prominent figure in Soviet era computer science and cybernetics. He worked on numerous inventions along the same time, working in parallel and often collaborating with several computer scientists in the United States and the UK. He's actually famous for likely writing the first optimizing compiler for a language that's more complex than Fortran. He wrote this for the alpha language that he worked on. And he was a really big proponent of education. In fact, he's probably the first computer scientist to talk about computer literacy. He built several schools during his time, he appropriated funding for them, and this, for example, is a photo at the USSR Academy of Sciences during a summer school for young programmers. His belief in teaching and in learning, and that being an important part of computer science, is really espoused in his speech titled Aesthetics and the Human Factor in Programming, which he gave at a 1972 computer conference in the United States. I'll read the full quote of which only a part is shown here. In past ages, the ability to read and write was considered a rare, God-given talent gift. The destiny of a limited group of the specially chosen. In the present age of general literacy, we perceive reading to be a universally attainable accomplishment, but we are tempted to single out a new elite group who become arbiters between the lay generality of mankind and the arcane informational model of the world hidden in the machine. Is it not, however, the highest aesthetic idea of our profession to make the art of programming public property, and thereby to submerge our elite exclusiveness within a mature mankind? Indeed, I feel like Urshav approaches these ethical problems slightly differently than some of his Western contemporaries. His idea is not, do we hold some power that we should then further hone for good or for evil. His point is, should we even hold this power at all? Should everyone learn to code? Should everyone learn how a computer works literally on the inside, not just how to turn it on and off? And if not, does our own understanding of computers, does our own ability to code, does, for example, the data scientists in the room, our understanding of statistics and machine learning models, does this give us some special power or privilege? And if it does give us a power or privilege, are we an elite? And what is being part of an elite mean? Does it mean we have a responsibility? Does it mean that we have to do things differently? Moving on to the networking era, which is always fun, right? We all love the Internet. Here you can see the employees at BBN, which is Bolt, Baranik and Newman, a Cambridge research firm. And BBN won a contract with the Advanced Research Projects Agency, otherwise known as ARPA. And their project was to build the first computer network in the United States. They're seen here in 1969 along with their interface messaging processor machine, which is essentially the first router. And in case you don't know, this is the same group that started to create the protocols and the standards that we use in today's Internet. There's a bunch of amazing engineers and architects that I could talk about in this photo, but I'm going to focus on two of them. Here in blue, I have several, and in yellow, I have Bob. Why just talk about the two of them? Well, first and foremost, because they went on to have longer careers and do quite a lot of work after ARPA. But secondly, because they were very outspoken in their political and ethical beliefs, and they went in quite different directions. So I think it's perhaps a good case study to explore the life of a computer scientist. We'll focus on Bobcon first. Bobcon worked on networks and essentially networking protocols. He along with Vincent Surf was able to create the TCP and IP protocols and standards, which are mainly unchanged today, which is pretty amazing, feet. He's essentially a networking genius. He went on to build very large networks for the US military at DARPA, and then after that he went to IPTO, which is the Information Processing Techniques Office. Then he heard about this new project brewing in Japan called the Fifth Generation Computer Project. And this idea was how can we create true AI thinking machines? How can we create these decision support systems that we might be able to use? At that time, he then pitched the idea of the strategic computing initiative, of which this is one of the plans. So for the strategic computing initiative, you have, as you can see at the bottom, all of the infrastructure. These are the networks that Bobcon loved. This is something that he loves to work on. He was able to get quite a lot of funding to just build networks and come up with new networking concepts. On top of that, they would then try to build some chips, some different hardware designs, and eventually working to natural language processing and speech and vision recognition and so forth. On top of those technologies, they would build autonomous systems, pilots associates, and battle management. And this was all to, and I quote, develop a broad base of machine intelligence technology to increase our national security and economic strength. So what does this mean? I mean, when I first saw this diagram, I said, well, perhaps Bobcon didn't even know about how his networks would be used. Maybe ethically for him, he just really wanted to work on networks. I can understand and appreciate that. And then I found a quote from around this era of Bobcon. And it states, the nation that dominates this information processing field will possess the keys to world leadership in the 21st century. As I began to read more of his writing during the time and writings of his peers at SCI, for example, Lynn Conway, who is a prominent computer scientist in her own right, it was clear that they were very well aware of how their programming would be used. In fact, it's rumored that the chart that we saw in the last slide, that Bobcon himself designed that pyramid. So it's clear that he knew he was building weapons. What I wonder is, what does it mean when we have to take military funding or let's say corporate funding of corporations, maybe we don't agree with, to do our work? Does this make us ethically culpable in what the greater process is? And if it does make us ethically culpable, if something goes wrong, let's say civilians are bombed and so forth, does that make you as the computer scientist or as the data scientist or the machine learning expert, does it also mean that you are culpable? And if you aren't culpable, then who is? Going back to Bobcon's peer, Severo Ornstein, he was a co-worker again and a peer at ARPANET. And here we see him with Laura Golt, who was a computer scientist and activist as well. Severo's work at ARPANET was as lead hardware expert. So he built most of the hardware for the first router and also did some of the software programming. He later went on to join Xerox PARC and he worked on the Dorado, which was the fastest processor at the time in operating system. So around the same time that Bobcon was petitioning the government for millions of dollars to build weapons, Severo Ornstein and Laura Golt were starting another initiative. And this was for a group called Computer Professionals for Social Responsibility. This is one of the patches that the Computer Professionals for Social Responsibility gave out at a joint AI conference, most likely the one in Los Angeles in 1985. And some older folks in the room or perhaps some that knew a little bit of Americana history. This is a play on a famous Cold War era advertisement, if you will. It was a public service announcement that came on and it would say, it's 11 p.m. Do you know where your children are? And its goal was to enforce curfew and to make sure that all the young people were inside at night in case of danger. Now this play on it is it's 11 p.m. Do you know what your expert system just inferred? And we see the telltale mushroom cloud of an atomic bomb. And these are the same expert systems that Bobcon himself was trying to build. I mean, they never really came too much, right? At that time it was very quickly before what we all know now as AI winter. And these expert systems were never much more than large statistical models. But the fear at the time was that they would be treated as experts. It was that if the system itself said, okay, we're under attack, it's time to launch the nukes, that that is something that would actually happen and that a human would go about doing it based on this expert system. That was a massive concern for CPSR, the computer professionals for social responsibility. Now CPSR was around actually until 2013. And they gave out fairly regular awards actually named Nelvarth Vina and highlighting people in the field who are working on social responsibility issues as computer professionals. It's a pretty amazing list. We've actually covered some of the people on the list, but it goes on to talk about whistleblowers and other people who really worked actively to act as an ethical agent. But I think what we can really take away from CPSR or what I took away from CPSR is that it was starting this type of activist conversation amongst computer professionals. And it was asking, what can we do as people who are concerned about the world, that as people who are concerned about social responsibility, how can we help? How can we be a part of change or a part of something that we believe brings justice to the world? And I think that it's quite sad when you read about the history of CPSR that around the time of the personal computing revolution in the mid to late 80s and early 90s, they saw their numbers dwindle. And I think one of the problems of this is computing went from being a social experience in the computer lab, working with their peers, chatting probably a lot about these types of issues and theories. It became a personal and a lone experience. This was also the time that is often tracked to the gender disparity within our field. That's often tracked that the advertisements were all of a sudden, be a gamer, it's really cool for boys to have computers. And so I feel like this was perhaps not intentionally, but unfortunately, one of the fatalities of that era was this conversation about computer professionals for social responsibility. But CPSR was not the only ones fighting for more professionalism in the field or fighting for us to be having these conversations. You might know of Karen Spock Jones. She's quite famous. She's a statistical, she's a statistician as well as a natural language processing expert. And she worked at the Cambridge Language Research Unit on a variety of NLP research. Probably her first forays, I guess, are into looking at inferring meaning from the Sari definitions in text. You might have also heard of TFIDF, term frequency, inverse document frequency. It's a popular way to extract important words or phrases from a text when you have a larger corpus. She wrote the paper that defined the IDF portion of the equation. So she's pretty important when it comes to natural language processing. But that's not her only contribution. Spock Jones was an outspoken critic of the gender disparities within our field. She also was quite a critic of not having some sort of professional code. She thought that the growing importance of computers in people's lives meant that we had more and more influence and she was concerned that we didn't have any type of licensing board review, that we didn't necessarily have any standards for our own code, our own actions. I'll again read a slightly longer quote. I certainly think that professionalism is very important. To be a proper professional, you need to think about the context and motivation and justifications of what you're doing. You don't need a fundamental philosophical discussion every time you put your finger on the keyboard. But as computing is spreading so far into people's lives, you need to think about these things. I've always felt that once you see how important computing is for life, you can't just leave it as a blank box and assume that somebody reasonably competent and relatively benign will do something right with it. For a quote, again, evokes this idea, who is responsible? If I build a program or write a script or build a model that does something unethical or that can be used unethically, am I responsible for when somebody does that? If I'm not responsible, who is? As especially machine learning and numerous other fields that we work in as they touch more and more lives, as they're used even by legal experts or by the police or by state intelligence or by doctors, are we not also beholden to these same licenses, these same ethics that these fields have? Now, I'm not necessarily saying we need to have a licensing or we need to have a particular code, but if we have no standards, if we have no way to review one another's work and to think about whether we're falling in ethical principles or whether we're helping or hurting lives, then we don't even have a starting place for these conversations on professionalism. And if we're not willing to take on the burden of thinking about these things, then does the burden actually go away or are we just ignoring it? Finally, we've reached some of the outspoken leaders of today's AI in today's era. This is Joanna Bryson. She is a professor of computer science who speaks on AI and natural intelligence at the University of Bath. And she's been working on the field of AI ethics since the 90s. She is a prolific writer and she's published all the time. She has a paper over this past year on the epics of looking at word vectors and different gender disparity within them. But she also has some older works. For example, her paper, Just Another Artifact, Ethics and the Empirical Experience of AI, articulates this problem of humans over identifying with artificial intelligence. She essentially says that instead of treating it like a book or a reference, we treat AI as a human actor, as another intelligent human being with reason and common sense and so forth. And she states that this is quite dangerous. If we start treating AI as more expert than ourselves, if we start treating it as the sole singular expert in the room, then what happens when it makes a mistake? So she's been quite outspoken on these issues. On one of her longer posts, she articulates some of her well thought out opinions on how we use data. I'll read again from a longer quote. As we in the computational social sciences learn more and more, our models of human behavior get better and better. As our models improve, we need less and less data about any particular individual to predict what they're going to do. So just practicing good data hygiene is not enough, even if that was a skill that we could teach everyone. My professional opinion is that there's no going back on this. But that isn't to say that society is doomed. Think of it this way. We all know that the police, the military, even most of our neighbors could get into our house if they wanted to, but we don't expect them to do that. And generally speaking, if anyone does get into our house, we are able to prosecute them legally and claim any damages back from insurance. I think our personal data should be like our houses. First of all, we shouldn't be seen as selling our own data, just leasing it for a particular purpose. This is the model software companies are to use for their products. We should just apply the same legal reasoning to we humans. Then if we have any reason to suspect our data has been used in a way we didn't approve, we should be able to prosecute. That is, the applications of our data should be subject to regulations that protect ordinary citizens from the intrusions of governments, corporations, and even friends. What Joanna Bryson is asking of us here is to actually be better than the current laws. Now we could get into some debates about the new EU privacy regulations and the fact that yes indeed it has some ways that I can control my own personal data and I can request information about how my data is used. But those are clearly not global and they have yet to be determined in the courts how they'll be enforced. What she's asking us to do is to treat people's data like their property. Would you buy and sell another person's property without them knowing it? And if not, why do we do this with customer data? Would you collect pieces of another person's property without properly informing them in plain English so that they can understand what they're agreeing to? And if not, why do we do it in our terms and conditions? And a particular concern of mine regarding security, would you take your friend's laptop or computer or books or whatever it is that they have and would you leave it on a public table outside for everybody to walk by and perhaps look at? And if not, then why are you leaving your databases with protected information or your models with protected information on the public internet? Debating ethics at work is nothing new. My great-grandmother was a pretty amazing woman. Here's her. Her name was Adele Matilda Rich. Her on the left is her growing up in New York City and she used to tell me really fun stories in the 20s and 30s of growing up in New York City. And her on the right is her with a very young version of me. Now, Adele, as I used to call her, was amazing. She has a lot of good memories that I have in my mind. But the reason why I bring her up is the fact that she worked as a secretary on the Manhattan project. The Manhattan project was the project in case you don't know the code words for the nuclear bomb in the United States. And she worked as a secretary. She used to tell me fun stories that she did something important for the war and that she had to burn her notebooks and so forth. And I just thought it was a fun story until I got older. She died when I was a child, so I was never able to have these types of conversations with her. But I did talk with my mother, who I know had several conversations. She said that Adele at first didn't know exactly what she was working on, but slowly put the pieces together over time and then of course when the bombs were dropped, she was quite aware. And I asked my mom, did she ever regret it? Did she ever stay up at night wondering if what she did was right or wrong? And my mom said that she thought that Adele felt like many of her generation that she was helping the good guys and that any deaths that were created by the atomic bombs dropped were just small casualties in a larger war. I wonder if I could go back and ask her if we could have a longer conversation about this. But it also makes me wonder if my own privilege now, something that she worked so hard to do. She worked to help make sure our family could survive through the war and the Great Depression. And she made it so that my mom could be the first one to go to college. And my mom going to college meant that I was able to get a computer and that I was able to go pursue computer science. So maybe it is wrong of me to question something like that. Perhaps she was not in a position to make a decision between ethics and feeding her family. And perhaps it is rather ignorant of me to even ask that question. But I also wonder if it is not a right for all of us to have rewarding work that's both challenging mentally, financially rewarding, and also ethically something that we can support. So I challenge us as a community. As data scientists, web developers, if you work on MicroPython, CorePython, Jython, whatever it is that you do, Internet of Things, I challenge us to start having these possibly more difficult conversations. I challenge us to start thinking about who is responsible and how we can hold one another accountable. Now, you might not work on anything that touches ethics, but perhaps you do. Or perhaps a coworker does, or a friend, or a colleague, or somebody you meet at this conference, perhaps a future you, or a past you. I know in my career I've been definitely asked to do unethical things. And I've been asked to lie, to make charts look different, to make data look different for the greater good, right, for extra funding, for whatever it is that the goal of the company or the product is. So I've had to have some of my own ethical decisions. I think that hopefully what we've heard today and what we've shared today can start to be a ground for a communal conversation. I know that in a community like this one, we have values. For example, I feel like as a Python community, we value diversity. I feel that we value free software and open source, and I'm very proud of that. I also feel welcome and supported today, just the same as my first Python conference in 2010. So we have these shared beliefs and these shared values already. But perhaps we don't have a larger understanding of how they relate to ethics in computing as a whole, or how they could inform a communal ethic, or communal principles, codes of code, and conduct, ideas that we share with one another, and conversations that we have openly. Now it may not be that we come to some communal agreement, but just the idea of sharing our stories openly, sharing stories like these from around the world openly that tell us a little bit about the history, the ideas, and these deep questions and dilemmas that we have. Perhaps that is a way to start being a better force in the world. We data scientists, programmers, Pythonistas, computer and software engineers, we do have this cultural and ethical history. I hope that it's shown this is only just a few of what I was able to find in my research. I want us to share these stories, and I want to hear your stories and have you share your stories as well of these ethical dilemmas that we face and the decisions that we make around them. And I truly, truly believe that in doing so, we are more. That when we have these conversations together, we are actually contributing to a greater good. Perhaps by having all of these conversations and working together, we can make sure that AI, machine learning, natural language processing, and Python have a bright future in justice. And that that's something that we can be really proud of. And I know that I want to be a part of that, and I hope that you do too. Thank you so much for being here, and I hope to continue this conversation throughout the conference. Thank you very much, Capric. I think it was an amazing keynote. Now everybody is really quiet because there's a lot of thinking going on in the room, but we have time for questions. So who has questions or comments? And if you're able, it would be nice if you could talk about that. Thank you very much. I have a question about, we'll just say, banks. There was no computer that they could call different. But thanks to the computers, we have automated ways right now, like PayPal, and the cryptocurrencies, which basically allows us to operate without banks. So what do you think about this? Yeah, so I definitely think I shared some folks today throughout history, and there was a few that touched a little bit on this, but I feel like this is definitely something more of our generation, our era, the idea of using kind of some of this technology to subvert the norm, right, to actually actively work against corporate influence on our life or government influence on our lives. So I think that there's a lot of ways that technology can be used to kind of actively work against some of these ideas. I completely agree, and I think that that's a really important conversation that unfortunately didn't quite fit in in this talk, but I think that's really important to share and think about. And this is kind of this idea of us creating technologies that actively support what we believe in, and then as we spread them in the world, that that hopefully makes more things possible. Like, I look at teams like the Signal team and what they're working on, and I think it's really great. There's some amazing work out there, and perhaps you work on those things as well, and these are really interesting conversations to me and things that we should also be highlighting, right? Thank you. Hi, thank you for the amazing talk. I would like to ask you a question about auto-automatization, especially if you see difference between what will happen probably with this machine learning revolution in some sense, and something like the industrial revolution with what happened in China, because the advantage that I see, or those other things that happened in the past, is that in fact they lifted a lot of people from poverty, even though it was not a great process in some sense, that was the main advantage. One should always say when you talk about, you know, auto-automatization machine, it has also human side, which was helping people. So do you think there is some differences between what will happen and what has already happened and why this should raise some different question regarding ethics? So, Zai, the question exactly is about how the computer revolution is different than revolutions before it, or, sorry. And the question is how do you think ethically this is different in some sense, one should be what we should think about? Is this just automatization on a larger scale, or is something, in a different way? Yeah, I wonder about that, because, yeah, clearly the industrial revolution was, and other previous revolutions, for example, the printing press and so forth created their own ethics and their own issues. What I wonder about with computers is, like as we've seen with Moore's Law, and then also with the new post-AI winter, we're seeing quite a lot of progress in a short amount of time. I don't know if that's the same amount of progress. I obviously wasn't alive then, and I'm no expert on previous revolutions. I'm not even an expert on this revolution. So I don't exactly know, but I wonder if the pace and perhaps the expanse is slightly more impactful now, that maybe because of the internet and because of the advances that we've made in computing, perhaps we have a more quick impact on people's lives. I don't know if this is true, but sometimes I think that perhaps this might be the case, but I would be curious to hear more of your thoughts on that topic as well. Great talk, really great. I was thinking about the ethical side of what we do on a daily basis is really important, but my concern is more, for example, the people who started investigating the outcomes and their subparticles many, many years ago, they had no idea that this will lead to the Manhattan Project. So do you have any advice on how we can try to predict the future implications of our work and how they can, the ethical result 50 or 100 years from now? This is something where if I had an answer for you, I would be telling it to the world from the rooftops. And I don't know, I think perhaps we need to, I echo that concern. I don't know what things that I work on today and things that people much brighter than me work on today. I have no idea what large impact these will have on the world a decade from now, even five or 10 decades from now. But I do think we should be thinking about it, and I think that I was heartened to when I was reading and doing research for this, that some of these people were trying to project and think about what this could do in the future. Now a lot of the times they were pretty far off. But I think that it doesn't hurt to start the conversation now and to work together to perhaps see what this might mean in 10, 50 years. And for example, how we can protect against things that we're fearful of and also support ideas that we think would be really great things that we can do with technology. Thank you for an awesome keynote. I think at least from my point of view, it's pretty obvious that automation of work is a trend that will continue and that will probably worsen any social disparities that we have now. Do you have any ideas where to start a conversation about how technologists can help in this situation instead of worsen the state that we are already in? Yeah, so I think that as I was thinking about this and when I found the computer professionals for social responsibility, now there are some groups that are working on these types of things. There's like fairness, accountability, and transparency, machine learning group and conference. There are some other ones that are trying to start these conversations about how we can be socially conscious actors when we do automation. I think that this would be a great thing to have a group around or a conference around or ideas around. I am no expert. I am just a person like you who is concerned. And I think that the more that we are able to share our ideas, I think that honestly, collectively, we are much more smart than any of us individually. And perhaps by having these types of debates openly, we can figure out maybe thoughtful ways that we can automate so that perhaps the impact is lessened. I also agree automation is definitely not going to go anywhere and nobody is going to stop automating because of a keynote or because somebody has concerns. So I think that, yeah, having these conversations and collectively coming up with some good solutions or some good ideas is most likely the most powerful thing that we can do. We have three more questions. And if you promise to keep them short, I'm going to take them. But I think probably since it's such an interesting subject, would you be willing to do an open space for that? Yeah, sure. Sure. So basically, if you don't know open spaces yet, we have a large room there. You can just join there and make up your own EuroPython. And there's a, yeah, that's what it's for. So there's a whiteboard downstairs. You can use it. So we're going to put an open space for this on. I'm going to tweet all also about it because I think many are interested and there's like three more short questions. Yeah. Yeah, cryptocurrencies have been just cited as one of the possible good signs of organization. I see them as a clearest example of what you said when you have technological advance without social advance because basically we have to burn oil because there was no other way than the proof of work to distribute bitcoins. Do you think this is related to some more intrinsic characteristics of computer scientists that this belief not just in current democratic institutions but in democratic processes in general, which makes it difficult maybe to have even just discussions because if they are pointless, we will not have them? Yeah. So you're asking like the anonymity that a computer allows me or that the internet allows me, does it break down these types of conversations? Yeah. So it's an anarchic background compared to the idea that democracy can help us have social advances. Yeah. I mean, this is a massive debate and something that I thought about including, but I just didn't have time to include like for example, these people that believe that we should have parts of the internet where you have to be yourself, right? Allowing us to have, let's say like use the internet as a system of political or social trust, right? And then you also then have governments using that to actively persecute activists and using the same like, okay, I can publicly identify you or I can easily de-anonymize you amongst big data so I can actively persecute you. And this is this like thing, the more that we create systems that we can use for perhaps creating these types of like public conversations where I can represent myself and on a global stage and be an identifiable actor is also a problem. And this is nothing new, right? This is something that since way before the internet, but it means that perhaps, yeah, in some cases, anonymization and these types of things are necessary to protect people. But I think it's a really interesting debate and there was a few people that I wanted to include that are having this debate openly about e-democracy and so forth. And I think that it's fascinating and it's something that, yeah, hopefully we one day can trust, let's say, international institutions to help support people if they come forward publicly and they identify themselves and they speak of atrocities in their country or so forth, Snowden, that they perhaps then are supported globally and have protections globally. And I just don't know if we're there yet, unfortunately. Okay, last question. So my question was a bit based on what you said right now. How exactly do you come out after you find something that's just seriously unethical, exactly like Snowden? He discovered that NSA was really not doing what it was supposed to do and you got to a point where, hey, you're just getting everyone's data and you're doing something that's incredibly more. How exactly do you come out? I mean, you start to dispose information that's not really for the public, but if you keep on doing what you're doing, you're actually compromising more. It's a very moral shaky ground. But your recommendation or your view. Yeah, so I think the protections for whistleblowers within computing, within internet and these types of things, we have to really evolve the conversation. That being a whistleblower for something like a massive multinational corporation or a massive multinational state intelligence agency, like these are things, where do you then go? Like who's supposed to protect you, right? And I think that this is particularly dangerous within our field. If I worked for a massive multinational corporation and I knew that we were doing something wrong, what am I supposed to do? If I come forward, I will most likely be fired. There will probably be a target of everything against my name, make me out to be an idiot, an unethical human and so forth. And if we as a community have more of ways that we treat whistleblowers and more ways that we protect whistleblowers, maybe people would feel more comfortable, right? Because I also agree that what are you supposed to do? And in some of these stories, I'll be publishing a little bit more about this and I'll be putting some stories on my blog of some of the folks and what they did. But for example, there is a man called David Parnas and he was a computer scientist and worked in Canada for the US Department of Defense along with a few other multinational defense agencies. And he wrote a big public resignation letter and he sent it all around everywhere he could think of. This is the days of the mailing list, not particularly public internet. And he publicly said, this is unethical. Everything that's happening within this organization is not something I believe in. I refuse to work here and I don't know why my peers continue to do so. And I think that we can look at some of these references in history to perhaps think about how we can make some of the same statements, but that also we have to figure out how to protect one another when one another is willing to step forward and do that. Okay. Thanks again for this amazing keynote.
history of computing with an ethical focus. So to talk about ethics in computing and what he saw instead to build tools and systems that we on social responsibility issues as computer actually covered some of the people on the list, thought that the growing importance of computers Pythonistas, computer and software engineers, we on a larger scale, or is something, in a days of the mailing list, not particularly public this organization is not something I believe in.